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NOTE  FROM  THE  MANAGING  EDITOR 


INTRODUCTION  OF  THE  NEW  DEFENSE 
ACQUISITION  REVIEW  JOURNAL  (flRJ) 


Since  the  inaugural  edition  printed  in  the  winter  of  1994,  the  Acquisition 
Review  Quarterly  ( ARQ )  journal  remains  loyal  to  its  original  intent  —  to  focus 
on  the  needs  of  acquisition  professionals  across  discipline,  serve  as  a  tool  for 
the  fostering  and  dissemination  of  scholarly  research  relating  to  the  acquisition 
community,  provide  a  platform  for  the  exchange  of  policies  and  opinions,  as 
well  as  create  a  forum  of  debate  and  discussion  on  issues  that  directly  affect 
the  acquisition  workforce. 

An  Acquisition  First 

As  the  first  publication  to  specifically  address  the  needs  of  the  acquisition 
workforce,  the  ARQ  continues  to  stay  current  with  a  rapidly  changing  Department 
of  Defense  (DoD)  from  transformations  in  the  Department’s  mission,  to  the 
establishment  and  implementation  of  new  business  processes.  And  in  light  of  these 
changes,  this  publication  is  making  a  few  changes  of  its  own  to  include  a  new 
name  —  the  Defense  Acquisition  Review  Journal  (ARJ)  —  and  the  adoption  of  a 
triannual  print  schedule  producing  three  issues  a  year  (April,  August,  and  December) 
versus  printing  four  issues  on  a  quarterly  schedule.  Please  be  advised,  there  is  no 
Fall  2003  edition  of  the  ARQ.  Instead,  all  efforts  went  toward  the  production  of 
the  edition  that  you  now  hold  in  your  hands. 


Our  Commitment 

We  as  members  of  the  Defense  Acquisition  University  Press,  stand  firm  on 
our  commitment  to  produce  a  premier  acquisition  publication  that  meets  the 
needs  of  the  acquisition  workforce,  and  I  invite  you  to  play  an  active  role  in 
helping  us  to  exceed  this  goal  by  submitting  manuscripts,  becoming  a  peer 
reviewer,  spreading  the  word  about  the  journal,  participating  in  surveys,  and 
writing  letters  to  the  editor  to  give  us  feedback  on  ways  that  the  journal  can 
better  service  the  DoD  Acquisition,  Technology,  and  Logistics  workforce. 
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Norene  L.  Fagan-Blanch 
Managing  Editor 
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LESSONS  LEARNED 


EVALUATING  THE  IMPACT 
OF  ELECTRONIC 
BUSINESS  SYSTEMS 

LESSONS  LEARNED  FROM 
THREE  CASES  AT  THE 
DEFENSE  LOGISTICS  AGENCY 


Jonathan  A.  Morell \  Ph.D. 

This  article  synthesizes  our  experience  evaluating  three  electronic  business 
(eBusiness)  systems  in  the  Defense  Logistics  Agency.  The  focus  was  on  actual 
impact  in  real  life  operational  settings.  We  summarize  our  experience  in  terms 
of  lessons  learned  and  make  a  case  that  our  experience  can  help  others  do 
similar  evaluation.  Lessons  learned  are  grouped  into  six  categories:  metrics 
and  data  sources,  methodology,  program  logic,  adaptive  systems,  realistic 
expectations,  and  dependencies  among  the  previous  five. 


This  article  synthesizes  our  experi¬ 
ence  evaluating  the  impact  of 
three  electronic  business  (eBusi¬ 
ness)  systems  in  the  Defense  Logistics 
Agency  (DLA).  Our  intention  is  to  show 
the  tactics  that  emerged  when  general 
principles  of  evaluation  were  applied  for 
the  context-specific  purpose  of  deter¬ 
mining  whether,  and  how,  an  eBusiness 
system  is  affecting  its  environment.  The 
first  section  outlines  our  emphasis  on 
impact  assessment  and  makes  a  case  for 
evaluating  eBusiness  systems.  The  sec¬ 
ond  section  presents  lessons  learned 


that  were  abstracted  from  our  experi¬ 
ences  and  that  can  be  applied  to  other, 
similar  evaluation  exercises.  Finally,  we 
illustrate  how  the  lessons  learned  were 
combined  to  produce  impact  assess¬ 
ments  of  particular  eBusiness  programs. 

Impact  Assessment  — 

Difficulties  and  Imperatives 


Our  evaluation  activities  assumed  that 
programs  that  have  been  deployed  should 
have  measurable  consequences.  In  this  we 
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are  firmly  rooted  in  the  tradition  of 
evaluation  for  impact  assessment.  This 
view,  as  summarized  in  a  classic  evalu¬ 
ation  textbook,  states  that: 


The  critical  issue  in  impact  evalu¬ 
ation,  therefore,  is  whether  a  pro¬ 
gram  produces  desired  effects 
over  and  above  what  would  have 
occurred  either  without  the  inter¬ 
vention,  or  in  some  cases,  with  an 
alternate  intervention,  (Rossi, 
Freeman,  &  Lipsey,  1999,  p,  239) 


Our  core  challenge  in  making  such 
an  assessment  was  the  need  for  a  meth¬ 
odology  that  could  produce  causal  in¬ 
formation  within  a  context  that  from  our 
evaluator’s  perspective,  was  totally  un¬ 
controlled,  We  had  to  evaluate  a  natural 
experiment ,  i,e,,  a  situation  in  which 
"...  program  variants  (or  other  treat¬ 
ments  of  interest)  are  not  experimen¬ 
tally  controlled  but  vary  in  the  natural 
environment  and  in  which  causal  infer¬ 
ence  is  still  desired.”  (Mark,  Henry,  Sc 
Julnes,  2000,  p,  265). 

In  the  present  case,  not  only  was  the 
situation  uncontrolled,  but  also  entirely 
post  hoc.  Evaluation 
did  not  begin  until  af¬ 
ter  the  programs  in 
question  were  well  es¬ 
tablished.  As  a  result  of 
the  timing,  it  was  im¬ 
possible  to  influence 
implementation  sehed- 
mmm  ules,  to  anticipate  data 
needs,  or  to  establish  data  collection 
mechanisms.  Of  necessity,  the  evalua¬ 
tion  design  was  quasi-experimental ,  an 
approach  defined  by  Rossi,  Freeman,  Sc 
Lipsey  (1999)  “An  impact  assessment 


in  which  ‘experimental5  and  ‘control5 
groups  are  formed  by  a  procedure  other 
than  random  assignment55  (1999,  p.  234). 
Data  limitations,  however,  made  it  nec¬ 
essary  to  formulate  tactics  that  went 
beyond  simple  comparisons  of  non¬ 
equivalent  control  groups.  Success  re¬ 
quired  knitting  together  many  dispar¬ 
ate  data  sources  and  analyses.  Much  of 
what  will  be  reported  below  is  the  story 
of  the  search  for  those  sources  and  the 
logic  and  methodologies  used  to  inte¬ 
grate  them. 

Because  of  our  emphasis  on  outcome 
assessment,  we  did  not  dwell  on  process 
metrics  such  as  percentage  of  time  a 
system  was  running,  average  time  to 
resolve  complaints,  or  number  of  users. 
Rather,  we  focused  on  whether,  because 
the  system  was  working,  there  was  mea¬ 
surable  impact  on  dollars,  quality,  time, 
or  readiness.  The  objective  was  to  deter¬ 
mine  whether,  for  operational  eBusiness 
systems,  it  would  be  possible  to: 

*  Obtain  relevant  data. 

*  Draw  conclusions  about  what  the 

program  accomplished. 

*  Develop  practical  recommendations  to 

facilitate  further  evaluation. 

The  answer  was  by  no  means  certain 
because  veiy  few  eBusiness  programs  are 
implemented  in  a  way  that  is  conducive 
to  impact  evaluation.  To  anticipate  the 
later  discussion,  limitations  on  IT  systems 
and  inter-organizational  agreements  con¬ 
spire  to  constrain  evaluation  possibilities. 
We  discovered  that  despite  these  prob¬ 
lems,  it  was  possible  to  assess  impact  for 
each  of  these  systems.  This  finding  gives 


evaluating  the  Impatt  of  electronic  Business  Systems 


us  confidence  (but  no  guarantee)  that 
impact  evaluation  can  also  be  con¬ 
ducted  on  other  operational  eBusiness 
systems.  By  presenting  this  information, 
we  hope  to  convey  a  sensibility  about 
how  this  kind  of  work  can  be  done,  and 
thus,  to  spur  more  such  activity  by  a 
larger  number  of  people.  At  the  DLA’s 
request,  three  eBusiness  systems  were 
studied:  Electronic  Document  Access 
(EDA),  Central  Contractor  Registration 
(CCR),  and  the  Department  of  Defense 
(DoD)  Email. 

EDA  (http://eda.ogden.disa.mil/ 
eda_main.htm):  The  Electronic  Document 
Access  Web  (EDA  Web)  combines 
Internet  and  World  Wide  Web  technolo¬ 
gies  with  electronic  document  manage¬ 
ment  to  eliminate  paper  files  and  facili¬ 
tate  information  sharing  among  DoD 
communities  to  provide  access  to  single¬ 
source  DoD  official  documents.  The 
information  is  maintained  and  available 
for  access  to  authorized  users  in  Portable 
Document  Format  (PDF).  Documents 
included  in  EDA  include  contracts  and 
contract  modifications,  MAAPR  (materiel 
acceptance  and  accounts  payable  report), 
government  bills  of  lading,  and  DD1716 
forms  (Contract  Data  Package). 

CCR  (http://www.ccr.gov/):  In  the 
past,  any  vendor  who  wanted  to  do  busi¬ 
ness  with  more  than  one  DoD  site  was 
required  to  submit  the  same  business 
information  to  each  and  every  site.  This 
redundancy  of  paperwork  not  only  cre¬ 
ated  an  administrative  burden  for  both 
the  government  and  the  vendor,  but  also 
was  a  major  source  of  administrative 
error  and  expense  in  terms  of  both  time 
and  money.  Because  DoD  is  the  largest 
purchaser  of  goods  and  services  in  the 
world,  the  cost  savings  to  be  incurred 


by  streamlining  these  administrative  pro¬ 
cesses  are  dramatic.  CCR  was  created  to 
be  the  single  repository  of  vendor  data 
for  the  entire  DoD  to  avoid  this  adminis¬ 
trative  duplication  and  allow  contractors 
to  take  responsibility  for  the  accuracy  of 
their  own  important  business  informa¬ 
tion  by  supplying  it  directly  to  the  gov¬ 
ernment  through  a  single  registration. 

DoD  Email  (https://emall.prod. 
dodonline.net/scripts/EMlogon.asp):  The 
DoD  Email  strives  to  be  the  single  en¬ 
try  point  for  purchasers  to  find  and 
acquire  off-the-shelf, 
finished  goods  items 
from  the  commercial 
marketplace  and  gov¬ 
ernment  sources. 

The  evaluation 
work  discussed  here 
took  place  between 
1999  and  2001.  The 
specific  findings  are 
frozen  in  time,  while 
the  programs  them¬ 
selves  have  been 
evolving.  Thus,  con¬ 
clusions  concerning 
the  systems  that  were  evaluated  may  not 
be  useful  for  current  decision  making. 
However,  we  believe  that  the  lessons 
learned  from  that  work  are  applicable 
to  evaluation  of  other  eBusiness  sys¬ 
tems  in  government  settings. 

Few  impact  evaluations  of  IT  systems 
take  place  in  government  settings.  But  to 
calibrate  expectations,  it  is  important  to 
realize  that  few  such  studies  exist  for  any 
sector.  Most  of  the  research  on  the  im¬ 
pact  of  IT  focuses  at  its  lowest  level  on 
the  firm,  and  aggregates  up  from  there. 
Much  of  this  research  deals  with  what  is 
commonly  known  as  the  productivity 
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paradox ,  i.e.,  the  disconnect  between  our 
intuitive  sense  that  IT  must  have  a  benefi¬ 
cial  impact,  and  the  failure  of  researchers 
to  observe  that  impact  (Brynjolfsson,  & 
Hitt,  1998;  Chan,  2000; 
Macdonald,  2002), 

A  second  body  of  re¬ 
search  on  IT  deals  with 
the  role  that  IT  plays  in 
particular  business  pro¬ 
cesses.  For  instance, 
Malone  and  Crowston 
(1994)  assess  how  IT  af¬ 
fects  inter-firm  transac¬ 
tion  costs,  and  by  so  do¬ 
ing,  influences  decisions 
about  trading  partner  re¬ 
lationships.  A  similar  fo¬ 
cus  is  exhibited  by  Argyres  (1999)  in  his 
research  on  how  IT  affected  inter-organi¬ 
zational  relationships  during  the  develop¬ 
ment  of  the  B-2  bomber.  Studies  like  these 
make  a  good  case  that  IT  can  play  an  im¬ 
portant  and  beneficial  role  in  shaping  de¬ 
cisions  about  how  an  organization  should 
behave.  However,  the  focus  of  most  exist¬ 
ing  research  and  evaluation  is  on  particu¬ 
lar  processes,  and  is  not  cast  in  the  form; 
“System  X  was  implemented.  What  im¬ 
pact  did  it  have?” 

The  reason  this  kind  of  evaluation  is 
difficult  is  because  when  specific  eBus¬ 
iness  systems  are  evaluated  within  a 
larger  organizational  context,  four  chal¬ 
lenges  to  good  measurement  and  good 
methodology  are  almost  always  present. 


"Hospiie  Hie  d  iff  i  - 
cutties,  impact 
assessment  of  DoD 
eBusiness  systems 
to  build 
wJh md  of  • 
knowledge, 
experience,  and 
wisdom  about 
whaf  works*" 


1 .  The  system  in  question  seeks  to  pro¬ 
vide  specific  and  limited  improve¬ 
ments  within  a  complex  context  of  mul¬ 
tiple  interacting  business  processes 
and  applications. 


2.  While  the  system  may  provide  spe¬ 
cific  assistance  to  a  well-defined 
group  of  users,  it  may  also  contribute 
to  an  overall  information  infrastruc¬ 
ture.  In  contributing  to  the  infrastruc¬ 
ture,  the  system  makes  additional,  and 
more  diffuse,  contributions  to  the 
development  of  other  systems  and  to 
creative  problem  solving, 

3.  At  the  same  time  the  system  is  being 
developed,  other  systems  may  also 
be  under  development. 

4.  Plans  for  impact  assessment  are  not 
put  in  place  during  the  programs5 
development  or  initial  deployment. 

Despite  the  difficulties,  impact  assess¬ 
ment  of  DoD  eBusiness  systems  is  needed 
to  build  a  fund  of  knowledge,  experience, 
and  wisdom  about  what  works.  As  this 
understanding  spreads  within  the  Do D 
system  development  community,  new  sys¬ 
tems  will  become  more  effective  and  more 
accountable. 

In  the  next  section  we  present  lessons 
learned  and  examples  of  their  application 
to  the  evaluations  that  were  conducted.  The 
subsequent  section  takes  a  deeper  dive  into 
the  EDA  evaluation  and  illustrates  the  les¬ 
sons  learned  in  greater  detail. 


Lessons  Learned 


Unambiguous  instructions  for  doing 
post  hoc  outcome  evaluation  are  impos¬ 
sible  because  evaluation  settings  differ 
with  respect  to  the  functionality  of  the 
system  being  evaluated,  comparisons 
that  can  be  drawn,  data  available,  user 
base,  and  implementation  schedules. 
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Accounting  Service  (DFAS),  an  organiza¬ 
tion  with  which  the  evaluators  had  neither 
personal  nor  contractual 
relationships. 

A  derivative  problem 
is  that  even  if  data  own¬ 
ers  are  willing  to  help, 
their  information  systems 
may  lack  the  capacity  to 
yield  the  fine-grained 
data  needed  to  evaluate 
a  particular  program.  Fur¬ 
ther,  no  matter  how  big 
an  organization,  any 
given  database  is  likely  to 
have  no  more  than  two 
to  five  people  who  un¬ 
derstand  the  database  in 
sufficient  detail  to  advise 
as  to  what  information 
can,  and  cannot,  be  extracted.  Moreover, 
the  identities  of  these  people  are  difficult  to 
ascertain  because  they  tend  to  be  organiza¬ 
tionally  distant  from  whatever  point  of  con¬ 
tact  an  evaluation  team  may  have,  and  also 
because  job  changes  often  necessitate  talk¬ 
ing  to  people  about  their  former,  not  their 
present,  jobs. 

The  above  problems  are  exacerbated  by 
the  fact  that  multiple  sources  of  data  are  likely 
to  be  needed.  Thus  efforts  to  find,  get,  and 
access  information  are  multiplied.  To  illus¬ 
trate,  consider  the  complexity  of  informa¬ 
tion  used  in  our  evaluation  of  CCR. 


"Further,  no 
matter  how  big 
an  organization, 
any  given  data¬ 
base  is  likely  to 
have  no  more 
than  two  to  five 
I  people  who 
understand  the 
database  in  suffi¬ 
cient  detail 
to  advise  as  to 
what  information 
can,  and  cannot, 
be  extracted." 


Collectively,  these  differences  are  bound 
to  have  major  consequences  for  choices 
about  design  and  analysis.  Rather  than 
be  prescriptive,  the  intent  of  this  section 
is  to  convey  a  sense  of  what  issues  must 
be  considered,  and  how  choices  might  be 
weighed,  when  deciding  on  how  to 
conduct  post  hoc  evaluation  of  eBusiness 
systems.  The  discussion  is  organized  by 
lessons  learned  in  six  general  categories: 

1.  Metrics  and  data  sources. 

2.  Methodology. 

3.  Program  logic. 

4.  Adaptive  systems. 

5.  Realistic  expectations. 

6.  Interactions  among  lessons  learned. 

Metrics  and  Data  Sources' 

All  relevant  metrics  are  categories  and 
combinations  of  dollars,  quality,  time, 
and  readiness.  The  challenge  is  to  de¬ 
fine  exemplars  of  these  metrics  such  that 
trusted  numbers  can  be  found  and  ana¬ 
lyzed.  One  major  problem  is  that  evalu¬ 
ation  is  usually  commissioned  by  a 
system’s  owners.  While  those  owners  can 
provide  rich  process  data  (e.g.,  number 
of  users,  up-time,  development  cost), 
they  usually  do  not  control  data  relevant 
to  impact.  Those  data  tend  to  be  owned 
either  by  a  system’s  users,  or  a  third  party 
data  collection  function.  To  illustrate, 
owners  of  EDA  believed  that  their  sys¬ 
tem  had  a  positive  affect  on  the  ability 
of  the  DoD  to  pay  invoices  on  time. 
Making  that  case,  however,  required  get¬ 
ting  data  from  the  Defense  Finance  and 


Relevant  information  came  from  the  De¬ 
partment  of  the  Treasury,  data  archives 
at  three  different  DLA  organizations,  and 
the  personal  knowledge  of  many  differ¬ 
ent  people. 

Electronic  Funds  Transfer  (EFT)  vol¬ 
ume  and  contract  transaction  volume 
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were  needed  to  construct  ratios  of 
actual  savings  to  real  savings.  To  do 
this,  two  different  sources  of  contract 
volume  were  helpful  in  improving 
estimation  accuracy. 

*  CCR  implementation  timelines  were 
needed  to  assess  the  likely  course  of 
events,  had  CCR  not  been  available. 
Transactions  costs  from  the  Treasury 
study  were  combined  with  transaction 
volume  data  to  assess  overall  impact. 

*  Qualitative  knowledge  about  OCR’s 
role  in  process  improvement  led  to 
a  logic  model,  which  dictated  the 
analysis  strategy. 

Methodology2 

Methodology  is  the 
logical  structure  in 
which  data  collection 
and  analysis  are  carried 
out.  Without  a  clear 
sense  of  that  logic,  there 
is  no  way  to  know 
what  to  do  with  metrics. 
For  instance,  an  evalu¬ 
ation  of  EDA  might  re¬ 
quire  using  the  metric  time  from  a  con¬ 
tract  being  finalized  to  its  arrival  at  the 
Defense  Contract  Management  Agency 
(DCMA).  But  how  should  this  metric  be 
used  to  draw  Inference  about  EDA?  Is  it 
necessary  to  track  the  metric  weekly, 
monthly,  or  annually?  Is  it  necessary  to 
compare  data  at  different  locations 
within  DCMA?  Is  there  a  need  to  differ¬ 
entiate  between  kinds  of  contracts?  Is  it 
necessary  to  obtain  historical  baseline 
data,  or  will  current  information  suffice? 
Would  it  be  beneficial  to  compare  con¬ 
tract  transmittal  time  to  DCMA  with 


transmittal  time  to  other  agencies?  An¬ 
swers  to  these  kinds  of  questions  make 
a  practical  and  significant  difference  for 
the  kind  of  evaluation  that  can  be  done. 

While  the  above  example  deals  with 
a  fine-grained  metric,  the  problem 
scales.  For  instance,  another  metric 
might  be  development  costs  for  IT  sys¬ 
tems,  to  be  measured  as  part  of  an  as¬ 
sessment  of  the  accomplishments  of  the 
Clinger-Cohen  Act.  There  is  no  doubt 
that  the  federal  government  has  many 
metrics  relating  to  the  cost  of  IT  sys¬ 
tems.  But  would  it  be  possible  to  com¬ 
pare  these  costs  over  a  twenty-year 
period?  Have  the  components  of  the 
metric  changed  over  the  years,  and  if 
so,  have  they  changed  in  a  way  that  in¬ 
validates  historical  comparison?  Or,  per¬ 
haps  different  federal  agencies  imple¬ 
mented  the  act  in  different  years.  Is  the 
time  difference  in  implementation,  com¬ 
pared  to  the  time  scale  of  the  metrics, 
conducive  to  comparison  across  agen¬ 
cies?  Would  the  data  allow  sub-depart¬ 
ment  level  comparison?  Depending  on 
the  answers  to  these  questions,  it  may 
or  may  not  be  possible  to  Implement 
different  evaluation  methodologies. 

Program  Logic 

Choosing  metrics  and  methodologies 
is  greatly  aided  by  developing  a  pro¬ 
gram  logic  model  in  order  to  answer  the 
question:  If  the  system  works  as 
planned,  what  will  be  different?  This 
may  not  be  an  easy  question  to  answer. 
A  program’s  impact  can  be  broader  than 
indicated  by  meeting  requirements  for 
well-defined  user  groups.  Proximate 
impact  may  induce  secondary  change. 
Time  frames  for  impact  may  vary  — 
some  changes  may  occur  immediately 


8 


Evaluating  the  Impart  of  Electronit  Business  Systems 


upon  system  implementation,  while 
other  changes  may  develop  over  years. 

Outcomes  may  interact  with  each 
other.  By  representing  these  phenom¬ 
ena  in  pictorial  or  tabular  form,  logic 
models  force  evaluators  to  identify  what 
to  measure,  what  measurements  to  com¬ 
pare,  and  when  data  analysis  should 
take  place.  Developing  these  models 
has  the  added  advantage  of  forcing  col¬ 
laboration  between  evaluators  and 
stakeholders,  and  in  achieving  consen¬ 
sus  among  stakeholders  as  to  what  out¬ 
comes  should  be  measured.  (The  field 
of  Evaluation  has  a  long  history  and  ex¬ 
tensive  literature  on  developing  logic 
models  to  drive  evaluation.  For  an  in¬ 
troduction,  see  Renger  and  Titcomb 
[2002].) 

Adaptive  Systems 

The  uses  of  eBusiness  systems  are  not 
static.  Of  course  all  such  systems  have 
core  uses  that  are  enshrined  in  require¬ 
ments  and  justification  documents.  These 
uses  represent  the  main  reasons  a  sys¬ 
tem  was  built,  and  their  evaluation  must 
carry  through  time.  Focusing  only  on 
these  uses,  however,  is  almost  certain  to 
miss  many  important  impacts.  (Whether 
these  are  desirable  or  undesirable  is  an 
empirical  question.)  Any  new  eBusiness 
system  represents  a  bundle  of  function¬ 
ality  that  constitutes  a  tool  people  can 
use  to  solve  problems. 

As  users  become  comfortable  with 
their  new  tools,  they  will  recognize  new 
uses  for  the  tools.  These  uses  cannot  be 
anticipated  because  experience  with  a 
tool  is  often  a  prerequisite  for  appreciat¬ 
ing  its  value.  Another  reason  is  that  per¬ 
sonnel  change  over  time  and  bring  new 
skills  and  new  perspectives  to  their  jobs. 


Additionally,  the  environment  in  which 
systems  operate  is  not  stable.  It  is  en¬ 
tirely  possible  that  by  the  time  a  system 
is  fully  deployed,  new  reasons  to  use  it 
will  appear.  (The  opposite  may  also  be 
true.  The  original  need  j 
for  a  system  may  dis-  § 
appear.  This  too,  must  I 
be  included  in  evalua-  I 
tion.)  I 

A  good  example  of  | 
newfound  use  is  the  I 
case  of  CCR.  CCR  was  I 
originally  conceived  as  | 
a  method  of  decreasing  | 
labor  for  data  input  by  I 
government  personnel,  1 
decreasing  the  number  1 
of  times  contractors  had  to  provide  the 
same  data,  and  increasing  data  accuracy. 
All  these  were  worthy  goals,  which  may 
have  justified  CCR.  However,  as  CCR  de¬ 
veloped,  its  true  power  came  to  be  real¬ 
ized.  For  the  first  time,  the  government 
had  a  single,  unambiguous  identifier  for 
all  government  contractors,  a  number  that 
remained  constant  and  reliable  across 
contracts  and  across  contracting  activities. 
This  ability  turned  out  to  have  major 
benefits.  For  instance,  it  was  instrumental 
in  facilitating  the  government’s  move  to 
electronic  payment  of  invoices. 

Realistic  Expectations 

One  of  the  most  frequent  questions 
evaluators  asks  is  some  variant  of:  “What 
are  your  expectations  for  what  this  sys¬ 
tem  will  accomplish?”  The  usual  answers 
are  almost  always  wildly  optimistic.  Per¬ 
haps  a  system’s  owners  can’t  get  out  of 
selling  mode,  or  perhaps  they  have  come 
to  believe  their  own  rhetoric  —  but  for 
whatever  reason,  claims  about  a 
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system’s  accomplishments  are  often  far 
beyond  any  reasonable  boundaries  of 
real  world  impact.  Woe  to  the  evaluator 
who  takes  these  statements  at  face  value 
and  proceeds  to  do  an  excellent  job  of 
measuring  the  program  relative  to  those 
projected  outcomes.  And  woe  to  the 
program’s  owners,  who  will  receive  only 
bad  news  about  the  value  of  their  efforts. 
The  disappointment  has  real  and  impor¬ 
tant  consequences. 

First,  program  managers  do  need  to 
justify  their  programs.  Evaluation  rela¬ 
tive  to  impossible  goals  will  not  provide 
that  justification.  Second,  program  man¬ 
agers  need  evaluation  data  to  help  them 
build  on  accomplishments.  Without 
knowledge  of  what  actually  happened, 
needed  guidance  is  missing.  Third, 
evaluation  almost  always  requires  the  co- 
operation  of  those  being  evaluated.  Over 
time,  assessment  that  brings  only  bad 
news  will  poison  the  climate  for  doing 
evaluation. 


While  almost  everyone  has  an  intui¬ 
tive  understanding  of  these  dynamics,  we 
have  found  that  the  generic  logic  model 
shown  in  Figure  1  is  extremely  useful  in 
driving  the  point  home  and  in  facilitating 
the  kinds  of  conversation  needed  to  iden¬ 
tify  measurable  achievements.  Figure  1 
depicts  a  program  made  up  of  four  pro¬ 
cesses.  An  eBusiness  system  is  imple¬ 
mented  for  the  purpose  of  lowering  the 
program’s  overall  costs.  Upon  close  in¬ 
spection  though,  it’s  obvious  that  the  new 
business  system  will  affect  only  Process 
Four. 

While  the  new  eBusiness  system  may 
improve  Process  Four,  it  may  not  change 
the  total  cost  of  doing  business  because 
mission  change,  or  high  level  reorgani¬ 
zation,  may  affect  the  scale  of  the 
program’s  activities.  Also,  changes  in  Pro¬ 
cess  Four  may  facilitate  other  internal 
changes  within  the  program.  Using  a  pic¬ 
ture  like  Figure  1  helps  get  stakeholders 
to  address  crucial  questions  about  scope. 
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Figure  1*  Realistic  Expectations 
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What  specific  process  will  be  affected?  If 
those  processes  were  improved,  how 
much  total  change  in  the  organization 
could  be  expected?  If  new  functionality 
became  available,  what  new  processes 
might  appear?  What  external  forces  are 
operating  that  might  affect  the  impact  of 
the  system  being  evaluated? 

Of  course  evaluators  must  not  cook  the 
books.  There  is  a  duty  and  an  obligation 
to  provide  accurate  information,  even 
when  that  information  will  work  to  the 
detriment  of  some  stakeholders.  Programs 
are  justified  to  funders  based  on  specific 
claims,  and  it  is  important  to  hold  man¬ 
agers  to  their  claims.  The  solution  is  to 
employ  a  variety  of  tactics.  First,  the  messy 
world  of  program  justification  and  devel¬ 
opment  is  a  web  of  political,  budgetary, 
and  bureaucratic  forces  that  requires  suc¬ 
cessful  managers  to  make  different 
claims,  in  different  ways,  to  a  variety  of 
groups.  While  some  of  those  claims  will 
be  core  justifications  that  must  be  evalu¬ 
ated,  others  will  not. 

Second,  eBusiness  systems  will  have 
intermediate  and  localized  impacts  that 
are  desirable,  and  that  provide  useful  feed¬ 
back  for  program  improvement.  These 
must  be  measured.  (Of  course  not  all  the 
local  or  intermediate  outcomes  may  be 
desirable,  and  these  too  must  be  assessed. 
Not  only  is  doing  so  necessary  for  a  fair 
evaluation,  but  the  information  can  also 
be  extremely  useful  for  designing  mid¬ 
course  corrections.) 

Interactions  Among  Lessons  Learned 

For  the  sake  of  exposition,  the  lessons 
learned  were  presented  as  if  each  were 
distinct  and  independent.  In  reality,  they 
are  inextricably  linked.  The  process  of 
evaluation  should  be  seen  as  a  continual 


scanning  for  these  relationships  as  the 
life  cycle  of  an  evaluation  unfolds.  A 
good  example  of  this  process  involves 
the  interaction  between  | 
data  sources  and 
methodology.  1-. 

One  of  our  early  jf 
plans  for  an  evaluation 
design  was  a  time  series  f 
analysis  of  a  particular  f; 
transaction  at  a  particular  f 
agency.  The  idea  was  to  ; 
compare  trends  before  f 
and  after  implementa-  ■ 
tion.  The  plan  seemed  k 
especially  appealing  be¬ 
cause  we  knew  that  the  system  had  been 
introduced  at  different  times  in  different 
parts  of  the  organization.  Our  team  was 
attracted  to  the  possibility  of  making  com¬ 
parisons  both  over  time  and  across  orga¬ 
nizational  subunits.  We  formed  this  plan 
because  trusted  informants  assured  us  that 
the  data  we  needed  had  been  collected 
over  a  long  period.  This  information 
proved  correct,  but  other  facts  emerged 
as  we  investigated  the  possibility  of  get¬ 
ting  that  data. 

First,  the  older  information  was  con¬ 
tained  in  a  system  that  had  been  phased 
out  and,  while  theoretically  available,  was 
not  obtainable  in  practical  terms.  Second, 
the  data  were  not  collected  at  frequent 
enough  intervals  over  the  several  years 
we  needed  to  provide  enough  data  points. 
Third,  the  way  in  which  a  critical  data  field 
was  defined  had  changed  over  time,  thus 
making  historical  comparisons  problem¬ 
atic.  Finally,  the  agency  itself  had  changed 
organizational  structure  over  the  years.  As 
a  result,  it  was  not  possible  to  compare 
change  over  time  either  within,  or  across, 
the  various  subunits.  In  light  of  these 


"The  process 
of  evaluation 
should  be  seen 
as  a  continual 
scanning  for 
these  relation¬ 
ships  as  the 
life  cycle  of  an 
evaluation 
unfolds" 
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discoveries,  it  was  necessary  to  abandon 
the  time  series  methodology  in  favor  of 
more  localized  comparisons. 

In  terms  of  the  practice  of  evaluation,  it 
is  important  to  note  that  our  initial  plan  was 
based  on  information  from  well-meaning 
people  with  good  knowledge  of  the 
eBusiness  system  involved,  the  agency  in 
which  it  was  used,  and  the  data  that  were 
generated.  However,  it  was  only  after  we 
had  a  chance  to  talk  to  many  mid-level 
and  lower-level  personnel  that  we  were  able 
to  get  the  specifics  needed  to  make  an  in¬ 
formed  judgment  about  whether  a  time  se¬ 
ries  methodology  was  practical. 


Applying  Ussons  Learned: 
The  Example  of  Electronic 
Document  Access 


Electronic  Document  Access  affects  so 
many  processes  that  a  wide  variety  of  in¬ 
put  was  needed  to  make  decisions  about 


logic  models,  metrics,  and  methods.  The 
organizations  whose  input  influenced  the 
EDA  evaluation  included  the:  Defense  Con¬ 
tract  Audit  Agency,  Defense  Contract  Man¬ 
agement  Agency,  Defense  Finance  &  Ac¬ 
counting  Service,  Defense  Information 
Systems  Agency,  Defense  Logistics 
Agency,  Directorate  for  Information 
Operations  and  Reports,  Fitting  Out  and 
Supply  Support  Assistance  Center 
(FOSSAC),  Navy/Air  Force  Interface,  Of¬ 
fice  of  the  Secretary  of  Defense  —  CIO 
Office,  and  several  Army  Commands, 

At  the  time  this  w?ork  was  carried  out, 
the  most  extensive  use  of  EDA  was  for 
the  management  of  contracts  and  contract 
modifications.  Thus  “contracts”  became 
our  primary  focus.  Potential  metrics  were 
cast  within  a  Balanced  Scorecard  frame¬ 
work  because  at  the  time  of  this  project. 
Balanced  Scorecard  was  being  heavily 
used  in  the  DL  A.3  We  felt  that  even  though 
our  work  was  unrelated  to  that  Balanced 
Scorecard  activity,  using  Balanced 
Scorecard  categories  wmuld  help  our 


fable  1.  Reasons  for  EDA  Impact 


Business 

Process 

Balanced  Scorecard 
Category 

Reason  why  EDA  may  be  Helpful 

DFAS  invoice 
processing 

Financial 

DFAS  requires  complete  paperwork  before  it  can  process 
an  invoice.  EDA:  1  -  reduces  time  from  document  creation 
to  its  arrival  at  DFAS,  and  2  -  assures  a  single  complete 
set  of  contracts  and  associated  modification.  The  result  is 
decreased  time  for  invoice  processing,  fewer  aged 
invoices,  and  better  compliance  with  the  Prompt  Payment 

Act. 

Contract/mod 

Financial 

EDA  has  the  potential  to  decrease  labor  effort  for  contract 

creation, 

distribution 

management,  and  as  such,  has  financial  implications. 

Internal  process 

Consistent  with  any  organization's  ability  to  adapt  to 
circumstance,  decreased  labor  effort  for  any  given  task  will 
result  in  a  reordering  of  work  priorities,  or  foe  development 
of  new  processes. 
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stakeholders  form  useful  linkages  among 
parallel,  but  conceptually  related,  activi¬ 
ties. 

Using  a  logic  model  perspective,  we 
articulated  why  EDA  should  affect  the 
metrics  identified.  The  mechanisms  of  ac¬ 
tion  are  presented  in  Table  1.  (Table  1  also 
illustrates  the  notion  that  while  logic  mod¬ 
els  are  usually  represented  in  graphical 
fashion,  tabular  descriptions  can  also  be 
useful.) 

EDA  Impact;  Contract  Processing  Labor 
and  Interest  Saved  on  Overaged  Invokes 

Data  used  in  this  analysis,  and  their 
sources,  appear  in  Table  2.  This  analysis 
again  illustrates  the  need  for  multiple 
sources  of  data,  some  of  which  reside  in 
data  archives,  and  some  of  which  were 


developed  for  a  specific,  empirical  in¬ 
vestigation  of  a  program.  In  the  present 
case,  the  data  came  from  FOSS  AC’s  de¬ 
tailed  and  careful  assessment  of  how 
EDA  affected  their  contract  process¬ 
ing  efforts.  For  their  contracts,  we  had 
good  information  on  labor  hours  and 
interest  payments  due  to  over  aged  in¬ 
voices  for  the  2000  and  2001  fiscal 
years,  i.e.,  the  time  immediately  before 
and  immediately  after  the  adoption  of 
EDA. 

The  limitation  on  the  FOSSAC  assess¬ 
ment  was  that  it  covered  only  a  small  num¬ 
ber  of  contracts.  To  scale  up  the  findings, 
it  was  necessary  to  determine  the  histori¬ 
cal  number  of  similar  paperless  transac¬ 
tions  for  the  whole  Department.  The  extra 
effort  to  determine  the  percent  paperless 


Table  2.  Data  Used  in  Assessment  off  EDA  Impact 
on  Contract  Processing  Labor 


Data 

Use 

Source 

Historical  data  on  DFAS  workload 

Contextual  understanding  of  how  DFAS 
worked,  the  pressures  operating  on  the 
Service,  and  why  better  access  might 
be  important. 

DFAS 

Per-contract  impact  of  EDA,  time 
before  and  after  EDA  implementation  at 
the  Fitting  Out  and  Supply  Assistance 
Center  (FOSSAC) 

Hard  data  on  change  due  to  EDA.  Used 
as  basis  for  scaling  up  estimate  to  the 
DoD. 

FOSSAC* 

Contract  volume  per  year  for  DLA, 

Air  Force,  Army,  Navy 

Used  to  scale  up  local  impact  to  DoD. 

1  -  OSD  CIO  Office 

2  -  DD350  database 

%  paperless  transactions 

EDA  only  contributes  to  change  for 
processing  of  paperless  transactions. 

"%  paperless"  is  needed  to  avoid 
applying  analysis  to  the  total  contract 
volume. 

1  -  OSD  CIO  Office 

2  -  DD350  database 

*  Data  courtesy  of  Bonnie  Brown-Murphy,  Management  Program  Analyst,  Fitting  Out  Supply  Support  Assistance 
Center,  Special  Projects.  Original  source:  Electronic  Commerce  Solutions  Corporate  Information  Management 
Board,  Paperless  Working  Group,  Oct  9, 2001. 
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Table  3.  Hours  Made  Available  Due  to  EDA 


Year 

98 

99 

00 

01 

02 

Total 

Hours/year 

3.8K 

42.4K 

49.8K 

51.7K 

51.7K 

250.9K 

was  critical  because  although  it  is  rela¬ 
tively  easy  to  find  the  total  number  of  con¬ 
tracts,  EDA  provides  labor  savings  only 
for  that  percentage  of  the  transactions  that 
were  paperless.  As  a  result,  two  data 
sources  had  to  be  used:  the  first  on  con¬ 
tract  transaction  volume,  and  the  second, 
on  paperless  transactions.  To  make  this 
determination,  two  data  sources  were  com¬ 
bined. 

The  first  was  information  on  total  con¬ 
tract  volume.  The  second  was  percent 
paperless  data  that  began  with  FY98  and 


ended  with  the  third  quarter  of  FYOL  Us¬ 
ing  all  this  information,  it  was  possible  to 
calculate  both  the  number  of  labor  hours 
that  were  no  longer  required  for  contract 
processing  due  to  EDA  and  the  savings  in 
interest  payments  due  to  EDA.  This 
information  is  summarized  in  Tables  3  and 
4,  Data  were  projected  several  years  into 
the  future.  We  ended  the  analysis  at  FY03 
because  while  projections  into  the  future 
are  legitimate,  the  further  the  projection, 
the  greater  the  inaccuracy.  Also,  we  had 
reason  to  believe  that  another  program  — 


Table  4. 

$M  Savings,  Interest  on  Overaged  Invoices  Attributable  to  IDA 


Agency 

FY97 

98 

99 

00 

01 

02 

03 

Total 

Army 

0.35 

0.79 

0.95 

0.95 

0.95 

0.95 

4.94 

Navy 

0.45 

0.99 

1.30 

1.30 

1.30 

5.34 

Air  Force 

0.44 

0.52 

0.52 

0.52 

2.00 

DLA 

7.14 

7.43 

7.44 

7.44 

7.44 

36.89 

Total 

0.35 

8.38 

9.81 

10.21 

10.21 

10.21 

49.17 

Cumulative 

0.35 

8.73 

18.54 

28.75 

38.96 

49.17 

ROI 

NPV  of  Savings 

43.31 

Investment  2,6  1.0 

1.0  1.0  1.0 

1.0 

0.5 

NPV  of  Investments 

7.33 

ROI  1 

5.91 

Notes: 

1,  Conservative  estimate.  Does  not  include  impact  on  discounts  earned,  bills  of  lading,  vouchers, 
MAAPR,  DD1716,  or  $  value  of  new  activities. 

2. 02/03  projections  based  on  00/01  data, 

3.  Unadjusted  $, 

4.  Return  on  Investment  (ROI). 

5.  Net  Present  Value  (NPV) 
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Wide  Area  Work  Flow  (WAWF)  —  would 
come  on-line  in  about  three  years,  at  which 
point  the  unique  impact  of  EDA  would  be 
blurred  by  the  combined  consequences  of 
both  programs. 

The  approach  taken  here  highlights 
possible  interactions  between  decisions 
made  about  metrics  and  decisions  made 
about  methodologies.  Our  initial  inclina¬ 
tion  was  to  find  one  or  two  metrics  that 
indicated  the  impact  of  EDA  and  that  could 
be  collected  on  an  organizationwide  ba¬ 
sis.  Had  we  been  able  to  do  this,  some  rela¬ 
tively  simple  comparisons  or  time  series 
analyses  would  have  sufficed  to  provide 
the  information  we  were  after.  Once  we 
learned  that  no  such  metrics  were  possible, 
we  began  to  cast  about  for  alternate  metrics 
and,  as  we  did  so,  for  methodologies  that 
could  exploit  those  metrics.  This  process 
led  to  the  tactics  we  actually  used,  i.e., 


we  took  a  micro-level  view  of  good  im¬ 
pact  data  and  brought  in  multiple  data 
sources  to  scale  up  the  findings  to  a 
broader  level. 

The  data  in  Tables  3  and  4  illustrate 
some  of  the  limits  that  must  be  accepted 
when  doing  post-hoc  evaluation  of  this 
type.  While  we  could  estimate  the  num¬ 
ber  of  hours  that  no  longer  had  to  be 
devoted  to  contract  processing,  we  were 
not  able  to  determine  how  organizations 
adapted  to  that  change.  Unanswered 
questions  included:  Did  they  decrease 
their  labor  force?  Did  they  reorganize? 
Did  they  deploy  the  workforce  to  other, 
truly  value  added  activities?  Any  of  these 
(in  multiple  combinations)  were  possible 
and  were  likely  to  vary  from  setting  to 
setting.  Because  no  mechanisms  were  in 
place  to  get  the  needed  data,  a  compre¬ 
hensive  evaluation  would  have  required 


Figure  2.  EDA:  Direct  Impact  and  Second-Order  Consequences 
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an  impractical  plan  that  far  exceeded 
available  resources.  First,  it  would  have 
been  necessary  to  identify  the  locations 
where  these  changes  had  been  taking 
place.  Second,  different  methodologies 
would  have  been  required  for  each  set  of 
outcomes. 

The  problem  of  data  access  is  a  practi¬ 
cal  limitation,  but  another  limit  touches 
on  the  fundamental  question  of  what  im¬ 
pacts  should  be  expected  from  any  given 
program.  To  understand  the  issue,  logic 
modules  can  be  of  assistance.  Figure  2 
illustrates  that  while  labor  hour  savings 


can  reasonably  be  expected  to  result 
directly  from  EDA,  the  follow-on  conse¬ 
quences  of  labor  hour  savings  are  affected 
by  powerful  forces  that  EDA  cannot  in¬ 
fluence.  The  immediate  impact  of  EDA 
is  that  as  people  start  to  use  it,  they  spend 
less  time  in  the  paper  processing  aspect 
of  contract  management.  But  what  hap¬ 
pens  once  the  time  is  saved?  There  could 
be  a  change  in  the  size  of  the  workforce, 
or  in  the  nature  of  the  organization,  or  in 
the  nature  of  work.  However,  none  of  these 
changes  are  direct  and  immediate  impacts 
of  EDA. 


Transaction 

processing 

WAWF 


Other  WAWF  impact,  not  part  of  this  system,  not  included  in 
this  evaluation 


Invoice  and  RA  processing  efficiency 

*  #  action  delayed  for  lack  of  contract  availability  to 
responsible  official 

*  #  manual  business  processes  eliminated  or  streamlined. 

*  #  FTEs  required  for  invoice  approval. 

*  #  FTEs  required  for  RA  approval 

*  #  unmatched  disbursements. 


Invoice  and  RA  cycle  time 

*  Time,  RA  receipt  -*■  approval. 

*  Time  delay  due  to  lack  of  contract  availability  when  needed 
by  responsible  official 

*  Access  time  for  contracts, 

*  Access  time  for  invoicing  and  receipt  data. 

*  Time,  detection  of  a  problem  resolution. 

*  Time,  invoice  approval  arrival  at  DFAS  for  payment. 


DFAS  cycle  time 

*  Notification  to  DFAS  payment  approval 

*  Delay  due  to  lack  of  contract  availability  when  needed  by 
responsible  official 

*  Access  time  for  contracts. 

*  Payment  approval  -*■  release  of  funds  to  vendor. 


DFAS  efficiency 

*  #  FTEs  required  for  payment  approval 

*  #  backlog  above  acceptable  level 

*  $  in  interest  payments  for  late  payment. 

*  $  discounts  lost  for  lack  of  timely  payment. 


TBD  for  each  system 


Total  time 
invoice 
submission  - 
(payment  receipt] 
for vendor 


Boundry  of 
assessment 
IF 

analysis  confined 
to  EDA 


Figure  3.  EDA  -  WAWF  Interaction 
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In  addition  to  the  operation  of  outside 
forces,  the  impact  of  any  single 
eBusiness  system  is  constrained  by  in¬ 
teractions  among  multiple  eBusiness  sys¬ 
tems.  In  any  large  organization,  many 
different  process  improvements  and 
eBusiness  implementations  are  likely  to 
be  under  way.  Any  single  system  is  part 
of  a  larger  developing  infrastructure. 
Change  in  multiple  parts  of  the  infrastruc¬ 
ture  is  needed  to  have  truly  profound  im¬ 
pact.  (Multiple  systems  are  also  the  root 
of  many  methodological  difficulties  be¬ 
cause  evaluation  requires  teasing  out  the 
impact  of  one  system  from  the  combined 
impact  of  several.)  The  need  to  limit  ex¬ 
pectations  for  any  single  eBusiness  sys¬ 
tem  is  illustrated  by  the  relationship  be¬ 
tween  EDA  and  WAWF. 

One  of  our  early  logic  models  (Fig¬ 
ure  3)  took  a  very  broad  view  of  EDA. 
In  doing  so,  it  included  the  expected  ad¬ 
vent  of  WAWF,  and  it  also  took  a  longer- 
range  view  of  likely  outcomes.  As  Fig¬ 
ure  3  shows,  EDA  alone  can  be  expected 
to  improve  internal  processing  efficiency 
at  DFAS.  DFAS  processing  time,  how¬ 
ever,  is  only  a  part  of  the  total  cycle  time 
from  when  a  vendor  submits  an  invoice, 
to  the  time  payment  is  received.  For  the 
entire  cycle  time  to  be  improved,  WAWF 
would  be  needed  to  shorten  many  other 
cycle  times  that  are  part  of  the  entire  pro¬ 
cess. 


Conclusion 


It  is  difficult  to  evaluate  the  impact  of 
eBusiness  systems  in  real  life  operation 


/  tr' 


because  the  data  needed  do  not  cleanly 
follow  the  contours  of  a  system’s  appli¬ 
cation.  This  is  true  both  organizationally 
and  temporally.  From  an  organizational 
point  of  view,  existing  data  often  cannot 
differentiate  those  parts  of  an  organiza¬ 
tion  that  are  using  a  system  from  those 
that  are  not.  From  a  temporal  point  of 
view,  data  may  not  be  available  over  time 
periods  that  will  allow  before  and  after 
comparisons  to  match  a  system’s  imple¬ 
mentation  schedule. 

Many  variations  onthese  themes 
exist,  and  many  prob¬ 
lems  derive  from  these 
difficulties.  For  instance, 
useful  data  may  be 
trapped  in  archaic  sys¬ 
tems.  The  definition  of 
data  elements  may 
change  over  time.  Be¬ 
cause  clean  data  cannot 
be  found,  multiple  data 
sources  are  needed  to  tri¬ 
angulate  on  a  conclu¬ 
sion,  and  the  greater  the 
number  of  data  sources,  the  greater  the 
likelihood  of  having  to  negotiate  with  re¬ 
calcitrant  data  owners.  Despite  these 
problems,  successful  impact  evaluation 
can  be  carried  out,  and  guidelines  —  les¬ 
sons  learned  —  can  be  abstracted  from 
past  efforts  that  are  applicable  to  future 
efforts.  (To  aid  in  this  application,  Table 
5  summarizes  critical  issues.)  We  hope 
we  have  convinced  the  reader  of  this 
conclusion,  and  that  by  so  doing,  spurred 
further  efforts  at  eBusiness  system  im¬ 
pact  assessment. 


iiill 


in  real  life  opera- 

■'"'1SS i 
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Table  5.  Critical  Questions  Within  Lessons  Learned 


Critical  Questions  Within  Lesson  Learned _ 

Metrics  and  data  sources 

What  data  are  needed? 

Who  owns  the  data? 

if  data  are  not  owned  by  group  that  commissioned  the  evaluation,  can  the  necessary  data  be  obtained? 
Who/where  are  the  few  people  who  truly  understand  how  needed  data  bases  are  constructed? 

Can  the  data  be  extracted  for  the  time  period,  and  at  the  level  of  granularity  needed  for  the  evaluation? 
Are  the  data  reliable? 

Methodology 

What  comparisons  can  be  made  to  determine  the  program's  impact? 

What  are  the  specific  targets  (e.g,  users,  business  processes)  of  each  comparison? 

What  are  the  threats  to  validity  for  each  comparison? 

Program  Logic 

Who  are  the  groups  that  must  agree  on  what  the  system  should  be  able  to  do? 

What  groups  and  business  processes  should  be  affected? 

What  are  the  proximate  and  secondary  impacts? 

What  elements  of  a  system  must  be  in  place  before  any  particular  impact  can  be  manifest? 

What  are  the  key  dependencies  in  the  system  and  among  impacts? 

What  are  the  time  frames  for  particular  impacts  to  appear? 

Adaptive  Systems 

As  a  system  becomes  known,  how  does  its  availability  affect  decisions  about  what  problems 
should  be  solved  or  opportunities  pursued? 

How  is  the  business  environment  affecting  beliefs  about  how  a  system  should  be  used? 

What  new  systems  are  being  implemented  that  draw  on  the  functionality  of  the  system  being  evaluated? 
As  new  uses  of  a  system  develop,  which  ones  are  important  enough  to  be  assessed? 

Can  evaluation  tease  out  the  contribution  of  one  system  from  another? 

Realistic  Expectations 

What  are  the  critical  claims  for  a  system's  value  that  must  be  measured? 

What  claims  on  their  face  are  unlikely  to  occur? 

What  reasonable  impacts  were  not  originally  envisioned  for  the  program? 

Interactions  Among  Lessons  Learned 

Does  development  of  the  evaluation  methodolgy  follow  a  "waterfall"  of  a  "spiral*  model? 

Is  there  a  process  in  place  to  detect  how  developments  within  one  lesson  category  may  affect  the  others? 
Does  the  evaluation  team  have  the  expertise  needed  in  qualitative  and  quantitative  methods  to  integrate 
an  evaluation  approach  across  all  lesson  categotries? 
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from  Northwestern  University  and  is  a  recipient  of  the  American 
Evaluation  Association’s  Distinguished  Service  Award.  Presently, 
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Endnotes 


1.  This  article  cannot  serve  as  a  com¬ 
plete  treatment  of  measurement  is¬ 
sues  in  evaluation.  For  a  good  in¬ 
troduction  to  this  topic,  see  Rossi, 
Freeman,  &  Lipsey  (1999), 

2.  As  with  the  topic  of  measurement, 
this  article  cannot  serve  as  a  com¬ 
plete  treatment  of  all-important  is¬ 
sues  in  evaluation.  For  a  good  in¬ 
troduction,  see  Rossi,  Freeman,  & 
Lipsey  (1999), 

3.  Balanced  Scorecard  in  an  organi¬ 
zational  planning  and  assessment 
approach  that  casts  leading  and  trail¬ 
ing  indicators  into  four  general  cat¬ 
egories:  financial,  customer,  internal 
business  process,  and  growth.  It  has 
been  adapted  for  other  contexts,  but 


the  principle  of  using  measures  from 
multiple  domains  is  consistent  Di¬ 
versity  of  measures  is  the  Balanced 
Scorecard’s  greatest  strength.  When 
a  single  overriding  metric  is  imposed 
on  a  system,  the  system  will  maximize 
that  metric.  Other  crucial  aspects  of 
organizational  functioning  will  be 
ignored,  thus  threatening  the  organ¬ 
ization’s  long-term  viability.  The 
power  of  the  Balanced  Scorecard  is 
that  it  helps  organizations  pursue  the 
joint  optimization  of  metrics  that  re¬ 
late  to  different  critical  domains.  For 
a  general  discussion  of  the  Balanced 
Scorecard,  see  Kaplan  and  Norton, 
(1996).  For  a  discussion  of  applying 
Balanced  Scorecard  to  information 
systems,  see  Martinsons,  Davison, 
and  Tse  (1999). 
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OPINION 


HOW  COMPENSATIOH  IN  TEST 
AND  EVALUATION  AFFECTS 
AIRCRAFT  ACQUISITION 

if  Col  lionet  D.  Alford,  USAF 

Systems  developers  and  testers  have  assumed  that  human  compensation  is 
measurable,  or  at  least  that  a  cognizant  and  trained  tester  is  able  to  identify 
and  detect  compensation.  More  than  one  study  conducted  at  the  Wright- 
Patterson  LAMARS  facility  indicates  that  this  is  not  necessarily  true.  Test  pilots 
were  able  to  compensate  sufficiently  to  fly  and  meet  defined  performance 
standards  on  intentionally  crippled  aircraft  flight  control  designs.  These  flight 
control  systems  were  designed  to  trigger  pilot  induced  oscillations,  but  in  most 
cases,  test  pilots  could  compensate  sufficiently  to  prevent  pilot  induced 
oscillations  and  to  control  the  simulated  aircraft.  Test  pilot  compensation  hides 
critical  handling  qualities  cliffs  that  can  lead  to  loss  of  an  aircraft  when 
encountered  by  less  skilled  pilots.  This  observation  has  vast  ramifications  for 
test,  evaluation,  and  development  of  all  human  interface  systems. 


After  solving  the  problems  of  pro- 
i  pulsion  and  lift,  the  control  of  an 
I  aircraft  was  the  third  and  possibly 
greatest  challenge  the  Wright  Brothers 
faced  in  conquering  the  air  (Figure  1). 
After  all,  the  Wright  Brothers  really  took 
eight  years  following  their  historic  first 
flight  to  determine  the  problem  of  stall  in 
a  turn  and  how  to  correct  it  (Culick,  2001). 
They  knew  how  to  control  the  aircraft, 
but  the  controls  were  insufficient.  The  air¬ 
craft  design,  power,  and  stability  were  all 
factors  in  solving  that  problem. 


Because  aircraft  design,  power,  and  sta¬ 
bility  all  affect  and  are  affected  by  the 
aircraft  control  system,  the  key  factor  in 
the  development  of  control  systems  for 
aircraft  is  to  design  them  to  optimize  the 
aircraft  performance  while  providing 
carefree  handling  qualities  to  the  pilot.  For 
example,  the  best  fuel  economy  is 
achieved  when  the  center  of  gravity  is 
behind  the  center  of  lift.  The  center  of  lift 
is  the  neutral  point  of  the  aircraft  and  hav¬ 
ing  the  center  of  gravity  at  or  behind  the 
center  of  lift  creates  a  condition  where 


DISCLAIMER 

The  views  expressed  in  this  article  are  those  of  the  author  and  do  not  necessarily  reflect  the  official 
policy  or  position  of  the  Air  Force,  the  Department  of  Defense,  or  the  U.S.  Government. 
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Figure  1.  The  Wright  brother's  first  flight  was  still  eight  years 
away  front  a  fully  stable/controllable  aircraft* 


the  aircraft  is  neutrally  stable  or  unstable 
—  both  bad  for  handling  qualities. 

An  aircraft  can  be  designed  with  an 
electronic  flight  control  system  that  gives 
the  pilot  a  positive,  stable,  handling-quali¬ 
ties  feel  while  the  aircraft  is  unstable  (Baer 
&  Landy,  1987).  This  design  results  in 
increased  fuel  economy,  a  characteristic 
of  the  Airbus  320+  and  the  Boeing  777. 
Future  airliners  will  certainly  capitalize 
and  expand  on  this  capability.  Another 
example  of  unstable  aircraft  design  to 
achieve  aircraft  requirements  is  found  in 
modem  fighter-type  aircraft.  For  radar 
stealth,  maneuvering  performance,  and 
mission  optimization,  among  other 
reasons,  the  exterior  of  military  aircraft 
are  designed  in  such  a  way  that,  without 
electronic  flight  controls,  they  would  be 
unflyable  (Rushby,  1993).  The  F-16,  F-18, 


F-22,  F-117,  and  B-2  are  all  examples 
of  this  type  of  design  (Rushby,  1993). 

The  problem  of  aircraft  handling  exists 
because  aircraft  controls  are  counter¬ 
intuitive.  Water  and  land-bome  transpor¬ 
tation  turn  using  a  device  like  a  rudder  to 
modify  the  velocity  vector  in  the  horizon¬ 
tal  plane.  This  is  an  intuitive  response  that 
is  easy  to  master.  To  turn  an  aircraft 
requires  a  roll  in  the  horizontal  plane 
coupled  with  a  pitch  rotation  to  counter 
loss  of  lift  in  the  vertical  plane  and  an 
increase  in  thrust  to  balance  the  increase 
in  drag.  The  pitch  rotation,  and  not  the  roll, 
turns  the  aircraft.  A  coordinated  turn  fur¬ 
ther  requires  a  corrective  yaw  rotation  in 
the  horizontal  plane  to  counter  the  slip 
induced  by  the  original  roll. 

Aircraft  motion  is  also  characterized  as 
a  mass-spring-damper  and  therefore  is  a 
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system  that  responds  in  a  manner  sig¬ 
nificantly  different  than  ground-based 
controls.  The  problem  of  aircraft  control 
in  thrust  and  pitch  is  further  complicated 
by  the  power  curve  response  in  the  region 
of  reverse  command  where  the  pitch 
control  largely  directs  airspeed  and  the 
thrust  directs  pitch  —  again  an  unintuitive 
response. 

People  can  generally  be  trained  to 
adequately  control  unintuitive  systems 
such  as  aircraft.  It  should  be  self-evident 
that  both  ground  and  flight  control  systems 
represent  natural  phenomena  that  are 
within  easy  grasp  of  human  beings. 
However,  higher-order  systems,  those 
greater  than  second  order,  are  not  gener¬ 
ally  found  in  nature  and  may  not  be 
predictable  by  human  beings  (National 
Research  Council,  1997;  Rushby,  1993). 

Modem  digital  flight  control  systems 
use  approximations  of  mathematical  equa¬ 
tions  that  result  in  very  high  order  sys¬ 
tems  to  replicate  the  natural  response  of  a 
non-electronically  controlled  aircraft. 
These  systems  of  high-order  approxima¬ 
tions  generally  do  a  great  job  of  reproduc¬ 
ing  the  handling  qualities  of  the  perfect 
aircraft;  however,  they  result  in  a  system 
that  potentially  is  unpredictable  to  the 
operator  and  they  introduce  unpredictable 
response  in  the  overall  aircraft  system 
(National  Research  Council,  1997). 

The  problem  of  unpredictability  of  a 
flight  control  system  is  characterized  by 
handling  quality  cliffs  and  pilot  induced 
oscillation  (PIO).  A  handling  quality  cliff 
is  an  unknown  and  untested  area  in  a  flight 
control  envelope  where  it  is  possible  for 
the  pilot  to  unexpectedly  lose  control  of 
the  aircraft.  A  PIO  is  a  situation  where 
aircraft  response  lags  the  pilot’s  input  to 
the  controls.  The  pilot  unconsciously 


"It  should  be  self- 
evident  that  both 
ground  and  flight 
control  systems 
represent  natural 
phenomena  that 
are  within  easy 
grasp  of  human 
beings/' 


increases  control  input  such  that  each 
input  magnifies  the  aircraft  response  until 
loss  of  control  or  the  aircraft  comes  apart. 
PIO  is  not  unique  to  digital  flight  control 
systems,  but  unforeseen  PIOs  are.  These 
problems  are  best  characterized  by  the 
distinctive  mishaps  they  have  spawned. 

On  October  26,  1977,  the  prototype 
Space  Shuttle  was  launched  from  its  747 
carrier  aircraft.  The  pilots,  Fred  Haise  and 
Gordon  Fullerton,  at-  “ 
tempted  a  spot  landing 
on  the  concrete  main 
runway  at  Edwards  Air 
Force  Base.  The  shuttle 
had  an  electronic  triply 
redundant  digital  fly-by¬ 
wire  flight  control  sys¬ 
tem.  The  expected  per¬ 
formance  of  the  aircraft 
did  not  match  the  actual 
performance  and  Haise  I 
found  himself  too  fast  on  the  approach. 
His  overcompensation  resulted  in  a  PIO 
in  roll  and  pitch.  In  spite  of  this,  he  landed 
the  shuttle  safely  (STS  Approach  and 
Landing  Test,  1977;  STS  Space  Shuttle, 
1977). 

NASA  engineers  found  a  270-millisec¬ 
ond  time  delay  in  the  flight  control  sys¬ 
tem  that  they  corrected  with  a  filter  (STS 
Space  Shuttle,  1977).  In  spite  of  this 
change,  the  pilot  astronauts  know  the 
shuttle  cannot  be  flown  like  a  fighter. 

The  flight  control  system  of  the 
shuttle  was  based  on  and  is  similar  to 
the  F-16.  The  F-16  had  and  has  a  known 
270-millisecond  time  delay  in  the  pitch 
axis.  If  the  aircraft  is  mishandled,  this 
delay  will  result  in  a  PIO,  and  PIO  has 
been  the  focus  of  numerous  mishap  in¬ 
vestigations  (Rushby,  1993).  The  result 
is  that  pilots  fly  an  F-16  approach  and 
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landing  like  they  would  a  heavy  aircraft 
and  not  like  a  fighter. 

Usually,  critical  PIO  problems  are  only 
identified  as  a  problem  when  they  cause 
an  aircraft  mishap.  On  February  2,  1987 
during  its  seventh  flight,  the  first  prototype 
Swedish  Saab  JAS39  Gripen  crashed  on 
landing.  The  Gripen  had  a  triplex  redundant 
fly-by-wire  digital  control  system  backed 
up  by  a  triplex  redundant  analogue  fly- 
by-wire  control  system.  The  first  test  pilot 
remarked  that  the  flight  control  system  was 
too  sensitive  and  displayed  problems  with 
lateral  and  pitch  oscillations.  The  pilot  fly¬ 
ing  during  the  mishap  had  never  flown  the 
Gripen,  and  gusty  wind  conditions  likely 
exacerbated  the  problems  with  lateral  and 
pitch  oscillations.  The  pilot  encountered 
increasing  PIOs  characterized  by  dynamic 
pitch  instability  during  approach.  These 
control  problems  resulted  in  the  aircraft 


striking  a  wing  on  landing  and  the  destruc¬ 
tion  of  the  aircraft  (Aviation  Week,  1989; 
Flight  International,  1989;  Nutley,  1989; 
Pellebergs,  1991). 

The  Gripen  program  went  through  a  very 
intensive  flight  and  ground  test  program 
to  fix  the  problems  caused  by  the  flight 
control  system,  and  the  aircraft  continued 
development.  Everything  appeared  fine 
until  on  August  8,  1993,  during  a  normal 
maneuver,  a  pilot  flying  the  Gripen  in  an 
airshow  fully  saturated  the  flight  controls 
and  entered  an  unrecoverable  PIO.  The 
manufacturer  and  the  customer  knew  that 
large  stick  movements  could  saturate  the 
flight  control  system,  but  the  pilot  was  un¬ 
aware  of  this  aircraft  characteristic.  The 
aircraft  was  destroyed  (Swedish  Accident 
Investigation,  1993). 

The  Gripen  is  not  the  only  aircraft  that 
has  experienced  interaction  of  the  pilot 


Figure  2.  The  Idlest  F-22  loss  of  control  incident  is 
possibly  a  digital  flight  control  cliff* 
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and  the  flight  control  system  that  re¬ 
sulted  in  the  loss  of  a  prototype  aircraft. 
On  April  25,  1992,  the  YF-22  (the  pro¬ 
duction  F-22  is  shown  in  Figure  2)  also 
crashed  during  landing  due  to  PIO 
caused  by  the  fly-by-wire  system 
(Rushby,  1993). 

Although  PIO  is  a  known  problem 
of  non-fly-by-wire  flight  control  sys¬ 
tems,  time  delays,  handling  qualities 
cliffs,  unpredictable  flight  characteris¬ 
tics,  saturated  control  systems,  the  at¬ 
tendant  PIOs,  and  loss  of  control  are 
characteristics  of  fly-by-wire  flight  con¬ 
trol  systems.  One  final  example  will  il¬ 
luminate  this  problem  of  PIOs  as  it  re¬ 
lates  to  testing.  Many  of  these  control 
problems  manifest  themselves  in  the  roll 
axis.  In  heavy  aircraft,  the  problems 
result  in  a  faster  roll  rate  than  normally 
expected  (Norton,  1994).  This 
unpredictability,  combined  with  trans¬ 
port  delays  cause  PIOs.  The  C-17  pro¬ 
gram  encountered  this  problem  late  in 
its  test  program.  The  aircraft  had  a 
known  quick  roll  rate,  but  test  pilots 
who  had  been  flying  the  aircraft  for  a 
while  did  not  consider  it  a  problem. 
New  test  pilots  generally  complained 
about  the  roll  rate  and  its  attendant  PIO 
during  landing  approach,  but  they 
quickly  learned  to  compensate.  When 
a  test  pilot  new  to  the  C-17  recognized 
the  problem  and  complained  offi¬ 
cially,  the  program  blamed  the  pilot 
and  continued  with  the  control  system 
unchanged.  In  Operational  Test  & 
Evaluation  (OT&E),  the  operational 
pilots  reported  the  problem  as  an  air¬ 
craft  deficiency  and  that  is  when  it  was 
finally  fixed. 

The  difficulty  in  the  C-17  program 
wasn’t  simply  that  a  handling  qualities 


problem  existed  in  the  roll  axis.  The 
problem  was  that  so  many  trained  test 
pilots,  military  and  civilian,  had  flown 
an  aircraft  with  an  obvious  deficiency 
and  found  it  acceptable  without  changes. 
The  factor  of  aptitude  that  allows  trained 
test  pilots  to  compensate  for  evident  de¬ 
ficiencies  in  flight  control  system  is  the 
problem  this  paper  directly  addresses. 


The  Experiment 


The  above  situations  demonstrate  that 
PIOs  in  fly-by-wire  aircraft  are  neither 
uncommon  nor  insignificant,  and  they 
are  more  likely  to  be  experienced  by  non¬ 
test  pilot  aviators.  At  best,  they  represent 
a  nuisance  and,  at  worst,  a  potentially 
catastrophic  air  event.  Because  of  these 
problems,  their  criticality  to  flight  and 
the  danger  of  not  finding  them  during 
developmental  testing,  we  are  studying 
PIOs,  when  possible,  through  simula¬ 
tions. 

In  December  1998,  I  participated  as  a 
subject  in  the  HAVE 
LIMITS  SIM  and  HAVE 
PIO  SIM,  PIO  study  con¬ 
ducted  by  the  Air  Force 
Research  Laboratory 
Air  Vehicle  Directorate 
(AFRL/VA)  that  used 
the  Large  Amplitude 
Multi-Mode  Aerospace 
Research  Simulator 
(LAMARS).  LAMARS 
is  a  20-foot  diameter 
sphere  on  the  end  of  a 
30-foot  beam  that  com¬ 
prises  a  5-Degree  of  Freedom  Simulator 
(Figure  3).  The  simulator  includes  a 
McFadden  Feel  System,  wrap-around 


"The  factor  of 
aptitude  that 
allows  trained 
test  pilots  to 
compensate  for 
evident  deficien¬ 
cies  in  flight 
control  system  is 
the  problem  this 
paper  directly 
addresses." 
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Error 


Figure  4.  The  heads-up  display  during  fhe  LAMARS  test 
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visuals,  and  a  Heads-Up  Display  (HUD)  resulted  in  a  departure  from  controlled 
(Figure  4).  The  LAMARS  system  is  flight.  In  most  cases,  the  pilot  could  not 
capable  of  up  to  -2  to  +3  g  vertical  and  respond  quickly  enough  to  go  open 

-3  to  +3  g  horizontal  acceleration.  loop  when  a  PIO  was  immanent.  Fur- 

The  purpose  of  the  experiment  was  ther,  as  reason  might  indicate,  the  higher 
to  gather  data  on  aircraft  handling  quali-  the  workload  of  the  task  the  easier  it 

ties  models  (good,  bad,  and  ugly)  to  was  to  depart  the  system.  This  was  not 

correlate  in-flight  variable  simulation  true  during  the  simulation  runs  with  ac- 
data  from  the  original  HAVE  LIMITS  celerative  motion, 
and  HAVE  PIO  open  air  programs  with  When  accelerative  motion  was  on,  the 
simulation  data.  The  experiment  fea-  tasks  became  easier  to  fly  as  the  workload 

tured  three  pitch  and  roll  capture  tasks  increased.  Pilot  compensation  and  leam- 

with  increasing  levels  of  workload  and  ing  occurred  at  a  rate  not  possible  with- 

a  landing  task.  The  aircraft  handling  out  motion.  Due  to  the  natural  feel  in  the 

qualities  models  varied  widely  based  on  acceleration,  it  was  increasingly  easy  with 

18  variations  tested  previously  in  the  increasing  workload  to 

CALSPAN/Veridian  NT-33  variable  sta-  maintain  control  of  the 

bility  aircraft.  The  study  itself  resulted  aircraft.  The  pilot  had  to 

in  good  data  but  simulation  results  force  himself  to  allow 

could  not  be  correlated  with  open-air  PIO  conditions  to  con- 

flight  test  (Stadler,  1999).  The  obser-  tinue.  It  was  very  easy 

vations  I  made  came  out  of  a  deviation  to  reduce  workload 

from  the  test  and  were  not  included  di-  slightly  and  allow  the 

rectly  in  the  test  or  research.  system  to  dampen  out 

I  had  the  opportunity  to  be  the  first  of  instead  of  pulling  ag-  h 

three  test  pilots  who  participated  in  the  gressively  to  the  point  that  would  have 

program.  When  I  arrived  to  make  the  first  departed  the  system.  This  was  very  ob- 

runs,  the  LAMARS  was  not  ready  for  vious  with  negative  G  during  pushovers, 

motion.  I  flew  a  full  set  of  the  simulations  Although,  the  simulations  felt  like  they 

as  training  without  simulator  motion.  A  were  often  on  the  ragged  edge  of  de- 

few  days  later,  I  flew  the  remaining  train-  parture,  it  was  possible  to  prevent  a  PIO 

ing  and  data  runs  with  LAMARS  accel-  and  a  departure.  The  feedback  I  re- 

erative  motion.  My  observations  come  ceived  was  that  the  departure  rate  over- 

from  the  unique  perspective  of  being  able  all  during  the  study  was  lower  than  ex- 

to  see  how  the  simulations  flew  with  and  pected,  “the  bad  was  not  as  bad”  as 

without  accelerative  motion.  seen  in  the  actual  aircraft  (Stadler, 

When  the  motion  was  off,  some  of  1999).  Additionally,  the  researchers  ob- 

the  configurations  were  impossible  to  served  that  pilot  anticipation  of  the  PIOs 

fly.  Many  of  the  flight  control  designs  may  have  skewed  the  data, 

were  divergent  and  resulted  in  complete  It  would  be  easy  at  this  point  to  con- 
loss  of  control.  The  simulator  was  rela-  elude  that  the  study  itself  did  not  pro- 
tively  easy  to  PIO  and  many  of  the  runs  vide  much  useful  data,  but  I  think  this 


"When 
accelerative 
motion  was  on.  Hie 
tasks  became 
easier  to  fly  as  the 
workload  in¬ 
creased." 
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program  highlighted  a  very  critical  area 
that  has  not  been  considered  much  in  flight 
testing  —  test  pilots  may  not  be  able  to 
gauge  how  much  they  are  compensating. 

Results 


The  observation  that  test  pilots  may  not 
be  able  to  gauge  how  much  they  are  com¬ 
pensating  is  not  as  obvious  as  it  seems.  We 
expect  average  pilot  subjects  to  not  know 
they  are  compensating  — 
we  assume  they  are 
compensating  and  we 
assume  we  can  mea¬ 
sure  that  compensation 
through  workload.  For 
test  pilots  the  situation  is 
different.  A  key  skill  in 
which  we  train  test  pi¬ 
lots  is  to  observe  and 
know  when  they  are 
compensating.  This  is  the  proverbial  test  pilot 
handshake.  If  test  pilots  cannot  gauge  their 
compensation,  then  there  is  little  hope  of 
solving  the  critical  problems  that  face  us  in 
digital  flight  control  systems.  Indeed,  based 
on  this  observation,  we  may  have  to  look 
for  a  different  way  of  designing  and  testing 
not  just  flight  control  systems  but  all  types 
of  human-machine  interfaces. 

As  long  as  the  aircraft  is  predictable,  and 
predictability  increases  dramatically  with 
natural  accelerative  motion,  the  pilot  can 
apply  reflexive  filtering  that  normally  pre¬ 
vents  PIO  and  departure.  As  my  experi¬ 
ence  in  the  simulator  with  and  without  ac¬ 
celerative  motion  demonstrates,  without 
accelerative  motion,  the  system  is  not  as 
predictable  as  with  accelerative  motion. 
Without  G  force,  the  system  is  less  pre¬ 
dictable.  The  aircraft’s  acceleration  makes 


possible  heroic  response  to  bad  flight  control 
systems. 

I  assert  heroic  response  with  intended  ex¬ 
perimental  precision.  Heroic  response  is  ex¬ 
actly  what  any  pilot  accomplishes  when 
faced  with  a  poor  flight  control  system  de¬ 
sign.  Experienced  pilots  unconsciously  feel 
the  natural/predictable  modes  of  an  aircraft 
and  successfully  compensate  for  poor  han¬ 
dling  qualities.  Most  pilots  do  not  realize 
the  degree  of  compensation  used  to  coun¬ 
teract  normal  aircraft  handling  qualities. 

In  an  aircraft  development  program,  as 
an  aircraft  flight  control  system  improves, 
the  test  pilot’s  compensation  improves  and 
without  a  significant  event,  such  as  a 
recalibration  of  the  pilot’s  compensation  ex¬ 
perience,  the  compensation  will  continue 
to  improve.  As  with  the  C-17  example,  and 
much  of  my  flight  test  experience  shows, 
test  pilots,  like  all  pilots,  will  at  some  point 
no  longer  be  able  to  gauge  their  compensa¬ 
tion  and  then  they  will  not  be  of  much  use 
to  the  test  program.  Without  training  or  com¬ 
parisons,  it  may  be  impossible  for  pilots  to 
gauge  the  degree  of  compensation,  espe¬ 
cially  with  long-term  programs  and  pro¬ 
grams  where  handling  qualities  have  im¬ 
proved  gradually  over  time. 

This  observation  is  true  of  flight  control 
systems  as  well  as  any  other  control  system 
in  an  aircraft.  I  further  suspect  that  this  ob¬ 
servation  concerning  compensation  and  test 
pilots  is  true  of  tests  in  all  other  complex 
systems. 

In  the  case  of  unnatural  or  unpredictable 
modes  of  digital  flight  control  aircraft,  these 
modes  can  only  be  learned  through  experi¬ 
ence  —  if  undiscovered  and  uncorrected, 
these  handling  qualities  cliffs  will  result  in 
loss  of  aircraft.  These  characteristics  of  pi¬ 
lot  compensation  make  digital  flight  con¬ 
trol  aircraft  more  difficult  to  sufficiently  test. 


"Acquiring 
firms  consider 
both  distressed 
and  highly 
profitable  firms 
as  potent ial 
acquisition 
candidates/' 
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Pilots  may  learn  to  compensate  to  the  de¬ 
gree  that  they  unconsciously  filter  even  un¬ 
natural  and  unpredictable  modes.  However, 
if  the  mode  is  not  experienced,  is  unpre¬ 
dictable,  or  is  not  discovered  and  corrected 
during  testing,  some  operational  pilot  will 
eventually  encounter  a  handlings  quality 
cliff,  and  recovery  may  not  be  possible. 

Recommendations 


1.  More  research  needs  to  be  accom¬ 
plished  on  measuring  pilot  compensa¬ 
tion  —  and  workload  may  not  be  a 
good  measure.  Workload  measurement 
has  been  the  Holy  Grail  of  human  fac¬ 
tors  testing.  To  date  we  do  not  have  a 
quantitative  measure  of  workload  and 
this  makes  human  factors  testing  sub¬ 
jective  and  difficult.  Quantitative 
workload  measurement  is  a  needed  and 
necessary  tool  for  human-machine  in¬ 
terface  development,  but  there  is  a  piece 
of  the  puzzle  that  is  still  missing  in 
workload  measurement  —  how  do  we 
quantify  compensation? 

2.  Test  pilots  require  hands-on  training  to 
understand  the  level  of  compensation 
possible  during  test  programs.  The 
LAMARS  facility  with  its  PIO  models 
provides  an  excellent  means  of  train¬ 
ing.  This  training  should  be  required  in 
every  Test  Pilot  School  curriculum  and 
taught  as  continuing  Test  Pilot  educa¬ 
tion.  Prior  to  the  test  of  digital  flight  con¬ 
trol  system  aircraft,  the  pilots  on  the 
program  should  attend  some  level  of 
refresher  orientation.  The  training 
should  allow  the  comparison  between 
seat-of- the -pants  accelerative  motion 
and  no-motion  to  drive  home  the  point 


that  too  much  exposure  to  bad  flight 
control  models  skews  the  pilot’s  per¬ 
spective,  and  a  pilot  can  become  too 
comfortable  with  a  poor  flying  system. 

3.  Test  pilots  need  to  constantly  recalibrate 
their  awareness  of  aircraft  handling 
quality  differences  and  compensation. 
The  best  way  to  achieve  this  is  through 
multiple  qualifications  and  qualification 
flights  in  different  aircraft.  All  programs 
could  benefit  from  this  regimen.  Test 
pilots  who  don’t  fly  multiple  aircraft  and 
who  cannot  compare  different  designs 
and  systems  lose  the  ability  to  identify 
their  level  of  compensation.  The  best 
method  to  keeping  this  critical  skill 
sharp  is  to  experience  known  deficient 
designs  and  poor  handling  aircraft.  Test 
pilot  schools  and  test  centers  should  en¬ 
sure  a  large  number  of  poor  aircraft  and 
historical  aircraft  are  available  for  test 
pilot  qualification.  The  services  should 
address  this  problem  by  allowing  test 
pilot  access  to  more  systems. 

4.  An  obvious  but  often  overlooked 
recommendation  in  the  analysis  of  fly¬ 
by-wire  systems  is  that  developers 
should  attempt  to  design  predictable 
flight  models  that  don’t  just  mimic  natu¬ 
ral  aircraft  response  but  truly  match  it. 

Conclusion 


We  may  have  underestimated  the  role  of 
compensation  in  testing  and  we  need  to  de¬ 
termine  ways  to  measure  compensation. 
Pilots  can  learn  to  adequately  fly  poor  air¬ 
craft  with  intentionally  poor  handling  quali¬ 
ties.  They  appear  to  be  able  to  unconsciously 
filter  certain  characteristics  in  the  handling 
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qualities  envelope  of  the  aircraft.  Unfortu¬ 
nately,  systems  developers  and  testers  have 
always  assumed  that  human  compensa¬ 
tion  is  measurable,  or  at  least  that  a  cogni¬ 
zant  and  trained  tester  is  able  to  identify 
and  detect  compensation.  The  HAVE  LIM¬ 
ITS  SIM  and  HAVE  PIO  SIM  studies  con¬ 
ducted  at  the  Wright-Patterson  LAMARS 
facility  indicate  that  this  is  not  necessarily 
true.  Test  pilots  were  able  to  compensate 
sufficiently  to  fly  and  meet  defined  per¬ 
formance  standards  on  intentionally 
crippled  aircraft  flight  control  designs.  This 
creates  critical  questions  for  the  testing  of 
future  human  interface  system.  To  help 
solve  these  problems: 

*  More  research  needs  to  be  accom¬ 
plished  on  measuring  pilot  compensa¬ 
tion. 

*  Test  pilots  require  hands-on  training  to 
understand  the  level  of  compensation 
possible  during  test  programs. 


*  Test  pilots  need  to  constantly  recalibrate 
their  awareness  of  aircraft  handling 
quality  differences  and  compensation. 

*  Developers  should  attempt  to  design 
predictable  flight  models  that  don’t  just 
mimic  natural  aircraft  response  but  truly 
match  it. 

Digital  control  systems  create  unique 
problems  for  engineering  design  and  flight. 
In  the  case  of  aircraft,  the  best  design  ap¬ 
proach  may  be  to  develop  predictable  flight 
models  that  directly  match  or  simply  aug¬ 
ment  natural  aircraft  response  instead  of 
using  complex  digital  equations  that  imi¬ 
tate  assumed  aircraft  response.  Until  that 
point  is  reached  and  because  of  the  diffi¬ 
culties  Involved  in  designing  human  inter¬ 
faces  and  the  human  control  of  complex 
systems,  we  must  find  quantitative  ways 
to  measure  compensation  and  we  must 
control  experiments  to  address  compen¬ 
sation  Issues. 
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TUTORIAL 


A  TEN-YEAR  REVIEW  OF 
THE  VISION  FOR  TRANSFORMING 
THE  DEFENSE  ACQUISITION  SYSTEM 

Edward  W.  Rogers,  Ph.D.  and  Col.  Robert  R  Birmingham i,  USA  (Ret) 

“With  this  report ,  then,  we  begin  a  decade-long  process  of  reinvention 

“We  hope  it  will  transform  the  habits ,  culture ,  and  performance 
of  all  federal  organizations . " 

(Former  Vice  President  A1  Gore,  1993) 

This  paper  traces  the  vision  for  reform  of  the  Department  of  Defense  Acquisition 
System  from  1993  through  2003.  Using  a  qualitative  document  review  process, 
a  conceptual  picture  of  overarching  themes  is  presented.  The  purpose  of  this 
paper  is  to  provide  an  analysis  of  the  change  roadmap  to  assist  building  empirical 
research  models  of  the  effectiveness  of  the  various  initiatives,  programs 
restructurings,  and  policy  mandates  that  have  all  contributed  to  the  current 
climate  for  change  within  the  DoD  and  the  acquisition  community. 


The  year  2003  marks  the  end  of  a  de¬ 
cade  of  concerted  effort  at  trans¬ 
forming  the  way  the  government  does 
business.  A  major  focus  of  that  effort  has 
been  the  transformation  of  the  way  the 
Department  of  Defense  (DoD)  acquires 
new  equipment  through  the  Defense 
Acquisition  System.  In  fact,  the 
beginnings  of  this  journal  were  steeped 


in  acquisition  reform.  The  topic  has  been 
an  on-going  subject  with  six  or  more  ar¬ 
ticles  per  year  dealing  with  acquisition  re¬ 
form  efforts.  The  inaugural  issue  in  1994 
opened  with  a  piece  by  then  Deputy 
Under  Secretary  for  Defense  (DUSD)  Col¬ 
leen  Preston  outlining  the  new  initiatives 
for  acquisition  reform  (Preston,  1994). 
The  pages  of  the  Acquisition  Review 
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Quarterly  have  proven  to  be  a  forum  for 
a  fertile  debate  on  the  merits,  means,  and 
misgivings  of  acquisition  reform.  This 
article  looks  back  over  the  last  ten  years 
and  traces  the  path  of  the  vision  for  that 
change. 

In  1993,  the  National  Performance 
Review  (NPR),  released  under  then  Vice 
President  Gore,  laid  out  a  vision  for 
change  that  many  have  considered  the 
landmark  for  a  new  decade  of  effort  to 
change  the  way  the  government  does 
business.  This  paper  at¬ 
tempts  to  take  a  con¬ 
ceptual  view  of  what 
evolved  from  that  NPR 
mandate  into  the  par¬ 
ticular  vision  for  chang¬ 
ing  the  Defense  Acqui¬ 
sition  System  within  the 
DoD  (Gore,  1993). 
What  makes  this  a  good 
time  for  reflection  is  the 
fact  that  near  the  end  of 
2002,  the  Deputy  Sec¬ 
retary  of  Defense,  Paul 
Wolfowitz  issued  a 
memorandum  cancel¬ 
ing  the  DoD  5000  series 
of  acquisition  policy  documents 
(Wolfowitz,  2002). 

Deputy  Secretary  Wolfowitz  judged 
the  latest  documents  to  be  “not  condu¬ 
cive  to  an  acquisition  environment  that 
fosters  flexibility,  efficiency,  creativity, 
and  innovation.”  This  raises  the  question 
of  what  exactly  happened  to  the  vision 
for  acquisition  reform  during  the  past  de¬ 
cade.  To  answer  this  question  the  authors 
of  this  paper  decided  to  bring  together 
their  respective  academic  knowledge  of 
organizational  change  and  the  practical 


experience  of  program  management  to 
examine  a  decade  of  reform  effort. 

Methodology 


We  conducted  a  broad  review  of  lit¬ 
erature  on  reform  of  the  DoD  to  identify 
key  documents  that  could  be  considered 
landmarks  or  mandates  for  the  acquisi¬ 
tion  transformation  process.  We  searched 
government  documents  available  on-line 
as  well  as  those  suggested  by  talking 
with  people  working  in  the  acquisition 
corps.  We  also  reviewed  published  ar¬ 
ticles  in  the  Acquisition  Review  Quar¬ 
terly  since  its  inaugural  issue  In  1994. 
From  a  total  of  several  hundred  docu¬ 
ments  collected  and  examined,  we  se¬ 
lected  seven  as  dealing  broadly  with  a 
vision  for  changing  the  acquisition  sys¬ 
tem. 

In  addition  to  the  document  search, 
Interviews  were  conducted  with  a  num¬ 
ber  of  individuals  in  the  Pentagon,  the 
acquisition  community,  program  man¬ 
agers,  and  industry  leaders.  The  purpose 
of  these  interviews  was  not  to  collect  a 
sample  of  data  but  to  clarify  the  under¬ 
standing  of  the  documents  and  the  in¬ 
tent  of  the  some  of  the  terms,  titles,  and 
statements  to  avoid  misinterpreting  the 
written  records.  We  also  used  the  inter¬ 
views  to  confirm  that  we  had  selected 
what  people  involved  in  defense  acqui¬ 
sition  generally  consider  the  landmark 
documents  that  have  set  the  direction  of 
change  during  the  decade.  Since  the 
overall  purpose  of  the  report  was  very 
broad  we  agreed  to  a  complete  non-at¬ 
tribution  arrangement  with  all  the  people 
interviewed. 
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We  distilled  the  contents  of  the  seven 
documents  into  tables  highlighting  their 
key  elements  for  ease  of  comparison. 
From  the  summarized  tables  and  inter¬ 
views,  we  traced  the  evolution  of  the  dif¬ 
ferent  aspects  of  the  overall  vision  for 
change  including  the  drivers  (perceived 
problems),  and  the  description  of  the 
desired  end  state.  From  our  analysis  of 
key  milestone  documents  we  developed 
a  conceptual  picture  of  how  the  trans¬ 
formation  vision  has  evolved  over  the 
decade.  Finally,  we  propose  several  ob¬ 
servations  that  should  be  addressed  with 
empirical  methods  to  help  answer  im¬ 
portant  policy  questions  regarding  ac¬ 
quisition  reform.  We  hope  this  review 
effort  will  help  spawn  research  and  de¬ 
bate  for  charting  the  way  towards  a  new 
and  better  acquisition  system  for  the 
DoD. 

The  National  Performance  Review  — 

1993 

The  decade  beginning  in  1993  cer¬ 
tainly  was  not  isolated  from  the  previ¬ 
ous  decades  and  prior  attempts  at  reform¬ 
ing  the  DoD.  David  Packard  had  a  large 
influence  on  the  reform  movement  start¬ 
ing  with  the  Packard  Initiatives  in  1969 
through  the  Packard  Commission  and  its 
report,  “The  Quest  for  Excellence”  de¬ 
livered  in  1986.  This  era  closed  with  Sec¬ 
retary  of  Defense,  Dick  Cheney’s 
“Defense  Management:  A  Report  to  the 
President”  in  1989.  These  efforts  at¬ 
tempted  to  deal  with  ballooning  costs, 
duplicative  programs  across  services, 
and  the  authority  lines  for  determining 
acquisition  priorities,  budgets,  and  pro¬ 
gram  evaluations  but  often  also  added 
layers  of  reporting  and  bureaucracy.  This 
led  Thomas  McNaugher  to  lament  at  the 


end  of  the  1980s  that  the  defense  acqui¬ 
sition  system  may  actually  be  worse  for 
the  reform  efforts  of  that  decade 
(McNaugher,  1990). 

Without  ignoring  the  impact  of  prior 
efforts,  we  limit  this  review  to  the  decade 
that  began  on  September  7,  1993  when 
Vice  President  A1  Gore  released  his  land¬ 
mark  report:  “Creating  a  Government  that 
Works  Better  and  Costs  Less:  The  Gore 
Report  on  Reinventing  Government”  as 
part  of  the  NPR.  While  the  Gore  Report 
primarily  focused  on  government  waste 
and  inefficiency,  elements  of  the  report 
impacted  efforts  at  trans¬ 
forming  the  way  the 
government  conducts 
the  business  of  defend¬ 
ing  the  country  as  well. 

While  the  Gore  report 
only  mentioned  three 
things  directly  related  to 
acquisition  reform,  the 
fact  that  the  Office  of  the 
Vice  President  was  be¬ 
hind  the  concepts  car¬ 
ried  considerable 
weight.  As  related  to  defense,  the  Gore 
Report  called  for  a  need  to  1)  simplify 
procurement,  2)  eliminate  regulatory  bur¬ 
den  and  3)  rely  more  on  the  commercial 
marketplace.  These  goals  spoke  of  a  need 
to  change  the  culture  of  how  the 
government  conducts  the  business  of 
defense. 

The  key  theme  of  the  NPR  was  that 
government  was  broken  and  the  system 
needed  to  be  overhauled  from  the  top  to 
the  bottom  to  regain  effectiveness.  After 
the  release  of  the  NPR,  many  people  in 
the  government  bureaucracy  faced  uncer¬ 
tainty  about  just  how  the  government 
would  function  in  the  future.  Such  a  broad 


"The  decade 
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change  mandate  coming  from  such  a 
high  level  created  a  sense  of  imminent 
change.  In  response,  many  departments 
began  to  develop  change  programs  in¬ 
cluding  the  DoD  and  its  new  Secretary, 
William  Perry.  The  release  of  the  NPR 
was  quickly  followed  by  a  document  out¬ 
lining  how  and  why  change  was  needed 
within  the  DoD. 

A  Mandate  for  Change:  The  Perry 
Memo— 1994 

On  February  9,  1994,  six  days  after 
being  confirmed  as  Secretary  of  Defense 
William  Perry  released  a  memo  titled,  “A 
Mandate  for  Change”  calling  for  a  com¬ 
plete  cultural  change  in  how  the  DoD 
operates.  Budgetary  and  efficiency  is¬ 
sues  remained  important  guiding  prin¬ 
ciples  drawing  from  the  NPR,  but  the 
focus  shifted  from  pro¬ 
cesses  to  outcomes  (ca¬ 
pability  in  the  field). 
The  main  directives  of 
the  Perry  Memo  were 
based  upon  themes  in 
the  NPR.  First,  he  called 
for  a  cultural  change 
pointing  out  that  the 
systems  themselves 
were  dysfunctional. 
Second,  the  outcomes 
were  not  effective  (pri¬ 
marily  too  slow).  Third, 
the  obstacles  to  change  were  mostly  in¬ 
ternal  (bureaucratic  inertia).  Secretary 
Perry  clarified  the  essence  of  the  need 
for  change  in  his  memo  and  created  the 
sense  of  urgency  that  required  immedi¬ 
ate  action.  It  also  became  clear  that  the 
DoD  was  not  exempt  from  the  NPR  man¬ 
dated  transformation  effort  in  the  Fed¬ 
eral  Government. 


If  the  NPR  showed  what  was  wrong 
with  government,  then  the  Perry  Memo 
demonstrated  what  was  wrong  with  the 
DoD  and  why  it  had  to  be  changed  now. 
With  many  examples  (following  the  NPR 
format),  Secretary  Perty  carefully  built  an 
argument  to  show  the  failure  of  existing 
systems  to  deliver  effective  solutions  to  the 
warfighters.  In  particular  he  noted  the  threat 
of  a  reduced  defense  industrial  capability 
in  the  post-Cold  War  era.  In  response  to 
the  challenge  laid  out  by  Secretary  Perry, 
an  office  was  created  to  specifically  deal 
with  transformation  issues  and  to  ensure 
that  change  was  made  in  an  effective  way. 

Acquisition  Reform  under  die  DUSD 
(AR )  Colleen  Preston  — 1995-97 

After  the  Perry  Mandate  in  1994,  a 
special  office  of  Deputy  Under  Secretary' 
for  Defense  (Acquisition  Reform) 
(DUSD[AR])  was  established  with  Colleen 
Preston  the  first  office  holder.  Her  vision 
for  change  was  based  heavily  upon  events 
at  die  time  including  the  recent  passage  of 
the  Federal  Acquisition  Streamlining  Act 
of  1993  (FASA),  The  formation  of  the  AR 
office  was  a  strong  signal  to  the  acquisi¬ 
tion  community  that  change  was  coming 
as  a  result  of  the  Perry  Memo.  Colleen 
Preston  became  a  spokesperson  writing 
articles  in  die  Acquisition  Review  Quarterly 
(Preston,  1994),  testifying  before  Congress 
(Preston,  1995a),  and  appearing  in  other 
DoD  publications  (Preston,  1995b). 

Her  mandate  was  to  find  ways  to  address 
the  declining  industrial  capability  while  im¬ 
proving  system  responsiveness  and  reduc¬ 
ing  costs  again  echoing  the  main  drivers 
of  the  Peny  transformation  vision.  It  thus 
was  a  mandate  for  both  increased  efficiency 
and  effectiveness  while  restructuring  the 
industry-government  relationship  base. 


"If  file  NPR 
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This  was  obviously  a  daunting  task  es¬ 
pecially  given  the  size  and  inertia  of 
the  defense  industry  and  the  acquisition 
community  in  general.  DUSD(AR) 
Preston  set  out  to  build  a  specific  change 
vision  around  the  elements  of  the  Perry 
Memo  by  importing  best  practices  from 
the  business  world. 

Three  specific  initiatives  probably  best 
characterize  her  efforts:  the  implementa¬ 
tion  of  Process  Action  Teams  (PAT),  the 
adoption  of  Integrated  Product  Teams 
(IPT),  and  efforts  made  to  capture  lessons 
learned  within  the  DoD  and  the  acquisi¬ 
tion  change  process  itself.  All  three  of  these 
have  endured  within  the  broader  acquisi¬ 
tion  community  as  acceptable  methods  of 
operation  throughout  the  decade. 

A  major  focus  was  directed  toward 
rewriting  the  DoD  Directive  5000.1  and 
the  DoD  Instruction  5000.2  documents. 
At  the  same  time,  a  project  was  started  for 
creating  desktop  PC  accessible  tools  for 
disseminating  the  new  policies  to  the  ac¬ 
quisition  community.  This  effort  involved 
collecting  best  practices,  success  stories, 
lessons  learned,  and  communicating  them 
as  widely  as  possible  across  the  acquisi¬ 
tion  community.  Practices  were  identified 
and  borrowed  from  industry  in  an  effort  to 
reduce  lead  time  and  cost  in  getting  state- 
of-the-art  technology  into  the  hands  of 
warfighters.  A  key  component  of  this  ap¬ 
proach  involved  using  commercial  suppli¬ 
ers  rather  than  defense-only  suppliers  to 
take  advantage  of  the  relaxation  of  military 
specifications  (MILSPECs)  and  the  policy 
implications  of  the  Federal  Acquisition 
Streamlining  Act  (FAS  A).  This  implied  fur¬ 
ther  shifts  in  the  defense  industrial  base 
requiring  fundamental  changes  in  the  con¬ 
tracting  relationship  between  the  govern¬ 
ment  and  its  key  suppliers. 


Defense  Reform  Initiative  —  Cohen 

1997-99 

At  the  end  of  1997,  acquisition  reform 
efforts  were  centered  in  an  office  called 
the  Defense  Reform  Initiative  (DRI).  Sec¬ 
retary  of  Defense  William  Cohen  in  the 
DRI  Report  in  November  1997  reiterated 
the  vision  and  urgency  of  continuing  to 
reform  the  way  the  DoD  conducted  busi¬ 
ness.  The  key  assumption  of  the  report 
was  that  under  constrained  resources  and 
new  threats,  existing  resources  must  be 
“reallocated  from  overhead  and  support 
activities  to  our  fighting  forces.”  DRI  also 
took  on  the  task  of  achieving  a  “Revolu¬ 
tion  in  Business  Affairs  (RBA)  to  support 
the  Revolution  in  Military  Affairs  (RMA) 
already  underway”  (Defense  Reform 
Initiative  [DRI]  Report,  1997,  p.  ii). 

Reducing  overhead  and  support 
structures  by  bringing  the  revo¬ 
lution  in  business  affairs  to  DoD 
will  be  critical  to  achieving  the 
revolution  in  military  affairs. 

(DRI  Report,  1997,  p.  ii). 

The  transformation  of  the  military  was 
driven  by  the  guidance  in  the  Joint  Vi¬ 
sion  2010  document  that  outlined  how 
the  future  military  forces  will  defend  the 
country.  A  central  element  of  that  vision 
is  maintaining  a  superior  ability  to  flow 
information  from  and  to  the  battlefield. 
This  information  flow  capability  struck 
a  chord  with  the  acquisition  community 
because  the  bureaucratic  jams  described 
in  the  NPR  and  the  Perry  Memo  pointed 
out  that  it  was  the  supply  and  support 
services  that  were  slowing  down  the 
military’s  responsive  capability.  The  new 
threats  would  not  allow  for  this  slack  in 
the  support  system.  The  acquisition  and 
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support  activities  must  be  brought  up 
to  speed  with  the  technology  and 
information  flows  to  the  field. 

This  vision  of  a  back-office  transfor¬ 
mation  to  support  the  field  operations 
became  an  identifiable  link  between 
acquisition  reform  and  military  effective¬ 
ness.  This  pointed  connection  may  have 
been  much  more  actionable  than  the  more 
general  government  is  broken  mantra  of 
the  NPR  and  Perry 
"Of  the  Memo.  While  a  boost  to 

many  policy  the  sense  of  urgency,  the 

and  structural  vision  also  took  on  a  pro- 

changes  mode,  grammatic  focus  in  the 

the  defense  sense  that  the  goal  of  re- 

Management  form  became  shortening 

Council  (DMC)  the  technology  develop- 

most  significant toe’ 

field  delivery,  and  doing 

both  at  a  lower  cost.  For 
example,  the  5000  rewrite  became  a  top 
priority  and  the  implementation  of  cost¬ 
saving  processes  the  critical  tasks.  In  the 
midst  of  this  process  improvement  the 
focus  on  fundamental  cultural  change  di¬ 
minished. 


Of  the  many  policy  and  structural 
changes  made,  the  Defense  Management 
Council  (DMC)  was  one  of  the  most  sig¬ 
nificant.  By  appointing  a  high  level  group 
representing  all  of  the  services,  the  intent 
was  to  bring  focus  and  momentum  to  the 
change  efforts.  The  decision  read  as  fol¬ 
lows:  “Establish  a  Defense  Management 
Council  (DMC)  to  serve  as  the  Board  of 
Directors  for  the  Defense  Agencies  and  to 
oversee  the  continued  reengineering  of 
DoDT  (DRI  Report,  1997,  p.  19),  The  goal 
was  to  get  senior  leadership  involved  in 
the  change  process  thereby  circumvent¬ 
ing  bureaucratic  inertia  and  resistance  to 
change  inherent  in  any  large  organization. 


While  it  is  clear  that  change  must  be  sup¬ 
ported  at  the  top,  the  momentum  for 
change  may  not  have  increased  with  the 
addition  of  the  DMC  as  a  monitor  of 
reform.  Being  a  representative  body,  it  may 
have  had  a  built-in  bias  for  protecting  the 
status  quo  rather  than  advocating  change. 

Thi  Road  Ahead  — 1999  (2000) 
Gansler  DUSD  (AT&L) 

The  next  major  direction  change  came 
in  1999  with  the  issuance  of  a  document 
called  The  Road  Ahead:  Accelerating  the 
Transfonnation  of  Department  of  Defense 
Acquisition  and  Logistics  Processes  and 
Practices 1  (Gansler,  2000).  This  document 
laid  out  the  framework  for  the  Revolu¬ 
tion  in  Business  Affairs  (RBA)  to  support 
the  term  Revolution  in  Military  Affairs 
(RMA). 

The  RMA  set  out  objectives  that  would 
have  to  be  met  in  order  to  ensure  military 
success  against  future  threats  and  en¬ 
emies.  The  RBA  was  meant  as  a  way  of 
stating  that  delivery  to  the  field  required 
a  different  management  of  the  acquisi¬ 
tion  and  logistics  supply  chain  behind  the 
new  military.  The  message  was  clear:  If 
the  military  had  to  change  to  meet  new 
threats,  the  support  system  had  to  change 
to  supply  the  new  military. 

The  DRFs  vision  documented  in  The 
Road  Ahead  clearly  shows  an  overall 
faster/cheaper  simplification  of  the  trans¬ 
formation  mandate.  There  was  much  more 
emphasis  placed  on  cost  issues  and  less 
focus  on  cultural  change  of  the  bureau¬ 
cracy.  The  three  goals  were  1)  faster  (re¬ 
duce  average  acquisition  cycle  time),  2) 
cheaper  (lower  total  ownership  cost  in  pro¬ 
gram  costs  and  logistics  support),  and  3) 
cheaper  (lower  overhead  costs  of 
acquisition  and  logistics). 
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Even  though  the  argument  of  cultural 
change  and  defense  industrial  base  sus¬ 
tainment  had  faded,  this  document  was 
widely  circulated  and  quoted  as  the  new 
direction  for  maintaining  the  change  mo¬ 
mentum.  This  document  brought  the  full 
force  of  the  change  effort  to  bear  on  re¬ 
ducing  costs  in  the  bureaucracy  thereby 
freeing  up  funds  and  time  to  improve 
deployment  to  the  field.  The  document 
gave  impetus  to  the  further  use  of  com¬ 
mercial  techniques  and  to  a  greater  use  of 
outcome  driven  performance  improve¬ 
ments.  This  meant  more  performance- 
based  contracts,  commercial  business 
practices  and  flexible  responsibility  at  the 
program  management  level. 

These  changes  had  a  profound  impact 
on  major  acquisition  programs  such  as  the 
RAH-66.  From  the  first  contract  award 
after  gaining  approval  to  proceed  into 
Demonstration  and  Validation  (MS  I)  in 
1991,  the  RAH-66  Comanche  Program 
underwent  a  series  of  directed  restructures 
as  DoD  priorities  shifted  during  this  re¬ 
form  decade.  With  each  restructure,  the 
entire  program’s  Training  and  Doctorate 
Command  (TRADOC)  approved  require¬ 
ment  and  contract  went  under  review  and 
updating.  Accommodating  the  changing 
acquisition  environment,  from  MILSPECs 
to  commercial  standards  coupled  with  a 
greater  reliance  on  performance  based 
contractual  agreements,  each  restructure 
adopted,  or  was  forced  to  adopt,  the  new 
acquisition  reform  initiatives. 

One  perhaps  unintended  result  was  that 
the  full  responsibility  for  total  system 
integration  via  performance  standards 
migrated  to  the  contractor  resulting  in  the 
perception  that  the  government  Program 
Management  Office  (PMO)  performed 
little  more  than  contractual  oversight  via 


participation  in  the  IPT  process.  The  para¬ 
digm  of  risk  ownership  migrated  away 
from  the  contractor  from  the  Lehman 
Years  Fixed  Price  Research  and  Devel¬ 
opment  (R&D)  contracting  methods 
where  the  burden  of  program  risk  rested 
primarily  with  the  con¬ 
tractor.  During  Pete 
Aldridge’s  tenure  as 
Defense  Acquisition 
Executive  (DAE),  he 
continued  the  transi¬ 
tion  of  risk  from  the 
contractor  to  the  gov¬ 
ernment  by  formally 
emphasizing  that  the 
burden  of  risk  in  the 
execution  of  complex 
development  and  inte¬ 
gration  programs  will 
be  placed  squarely  on 
the  shoulders  of  the 
government. 

Achieving  Key  Performance  Param¬ 
eters  (KPPs)  was  no  longer  a  decision  of 
which  engine,  gun,  or  rotor  system;  but 
how  much  shaft  horsepower,  how  many 
rounds  per  minute,  and  what  rate  of 
climb.  For  example,  the  initial  weight  and 
cost  goals  for  Comanche  as  directed  from 
then  Army  Acquisition  Executive  (AAE) 
James  Ambrose  in  1987,  was  for 
Comanche  to  weigh  no  more  than  7,500 
pounds  and  cost  no  more  than  $7.5  mil¬ 
lion  per  aircraft.  Although  noble  goals, 
they  were  not  performance-based  objec¬ 
tives.  However,  regardless  of  the  acqui¬ 
sition  reform  methods,  these  two  goals 
continue  to  haunt  the  program  as  mea¬ 
sures  of  how  much  the  aircraft  perfor¬ 
mance  has  changed  and  degraded  over 
time.  This  raises  an  important  reform 
question  of  which  standards  will  be  used 
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to  measure  reform  progress?  These  is¬ 
sues  demonstrate  the  need  for  clear  vi¬ 
sion  of  the  end  state,  not  just  the  sav¬ 
ings  or  speed  that  the  process  changes 
hope  to  achieve.  What  is  the  vision  for 
how  weapons  systems  will  be  acquired 
in  the  future?  How  will  programs  cur¬ 
rently  in  the  pipeline  be  measured  when 
they  were  started  under  one  system, 
modified  numerous  times  as  the 
Comanche  was,  and  perhaps  finished  un¬ 
der  a  third  acquisition  environment? 

The  Road  Ahead  was  also  the  launch¬ 
ing  document  for  the  third  reform  program 
office  under  the  auspices  of  the  DUSD  Ac¬ 
quisition,  Technology  and  Logistics 
(AT&L)  Pete  Aldridge 
called  the  Office  of 
Acquisition  Initiatives 
(AI),  The  AI  office 
continued  to  work 
through  2002  on  a  num¬ 
ber  of  initiatives,  the 
major  one  being  to  re¬ 
write  the  5000  series  of 
documents  guiding  ac¬ 
quisition  policy.  Other 
efforts  included  a  new 
guide  to  managing  intel- 
lectual  property,  pro¬ 
gram  risk  management, 
and  contracting  tools 
(cost  as  an  independent  variable,  earned 
value  management  and  performance 
base  contracting,  etc.).  The  Office  of  the 
Secretary  of  Defense  (OSD)  AI  worked 
to  document  reform  progress  and  cost 
savings.  The  office  also  worked  hard 
to  get  the  new  information  disseminated 
to  the  acquisition  community.  A  desk¬ 
top  reference  was  put  together  allow¬ 
ing  on  line  access  to  many  of  the  policy 
and  guide  materials  for  acquisition. 


The  Defense  Acquisition  University 
(DAU)  was  reorganized  and  decentralized 
into  five  regional  campuses  to  get  closer 
to  the  customers.  Many  of  the  5000 
changes  appeared  as  new  directives  for 
program  managers’  compliance.  Faced 
with  a  growing  list  of  prescriptive  com¬ 
pliance  items  to  document  adherence  to 
reform,  some  program  managers  began 
to  view  these  reform  efforts  as  burdens 
rather  than  process  improvements.  Real¬ 
izing  the  reform  cycle  had  come  full  circle 
to  where  it  was  creating  the  additional 
paperwork  it  was  supposed  to  eliminate 
might  have  contributed  to  some  of  the 
cynicism  toward  acquisition  reform  in  the 
program  offices  during  this  later  part  of 
the  nineties. 

For  example,  the  burden  to  accommo¬ 
date  the  best  practices  in  measuring  earned 
value  of  work  performed  within  the  con¬ 
fines  of  a  large  R&D  program  was  par¬ 
ticularly  difficult  as  more  and  more  con¬ 
tracts  were  awarded  to  partnerships  and 
Joint  Ventures.  Few  realized  how  difficult 
a  task  it  would  be  to  merge  two  completely 
different  cost  accounting  paradigms  into  a 
single  Earned  Value  Management  System 
(EVMS)  document.  For  Comanche  and 
other  large  complex,  and  long  running  pro¬ 
grams,  the  process  was  even  more  com¬ 
plex  as  the  mergers  and  acquisitions  of 
major  Defense  contractors  brought  the 
entiie  DoD  helicopter  industry  dangerously 
close  in  terms  of  labor  and  overhead  rates. 

Additionally,  as  Comanche  underwent 
a  series  of  four  major  restructures 
throughout  the  1990s,  measuring  earned 
value  from  a  realistic  and  established 
baseline  was  seldom  more  than  a  two- 
year  event.  It  was  no  surprise  that  Boeing 
and  Sikorsky  enjoyed  EVMS  metrics, 
which  seldom  deviated  from  1  over  a 


"Faced  with  a 
growing  list  off 
prescriptive  com¬ 
pliance  items  to 
document  adher¬ 
ence  to  reform, 
some  program 
managers  began 
to  view  these 
reform  efforts  as 
ha rdens  rather 
than  process 
improvements*" 
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ten-year  period.  It  was  not  until  the  MS  II 
contract  award  that  the  EVMS  data  being 
reported  indicated  that  neither  contractor 
could  keep  up  with  the  planned  work  of 
funding.  Although  criticized  for  often 
shifting  the  baseline  work,  the  EVMS  pro¬ 
cess  implemented  by  the  Comanche  con¬ 
tractors  provided  a  clear  picture  as  to  how 
far  behind  schedule  and  over  budget  the 
program  was  headed  just  six  months  af¬ 
ter  the  contract  was  awarded  in  2001. 

During  the  most  recent  restructure  of 
Comanche,  the  Program  Manager  en¬ 
forced  contractual  requirements  for  the 
Joint  Venture  to  implement  an  integrated 
management  plan  that  accommodated 
differences  in  manpower  loading,  over¬ 
time  rates,  and  labor  rates  across  the  two 
companies  to  reflect  more  accurate  and 
timely  EVMS  information.  These  are 
examples  of  how  reform  efforts  often 
require  more  attention  than  imagined  to 
implement  and  have  consequences  in 
industry,  labor,  and  commercial  markets 
that  affect  the  outcome  of  initiatives. 

Rumsfeld's  Vision  —  September  10, 

2001 

The  events  of  September  11,  2001 
raised  dramatically  the  urgency  of  solv¬ 
ing  acquisition  problems.  Ironically,  on 
September  10, 2001,  Secretary  of  Defense 
Donald  Rumsfeld  delivered  a  key  speech 
outlining  his  determination  to  liberate  the 
Pentagon  from  itself  by  reducing  bureau¬ 
cracy  and  simplifying  the  acquisition  pro¬ 
cess.  This  document  laid  out  the  new 
secretary’s  vision  for  building  the  future 
military  and  the  defense  system  to  sup¬ 
port  it.  The  system  was  the  enemy  and 
needed  to  be  defeated.  “The  topic  to¬ 
day  is  an  adversary  that  poses  a  threat, 
a  serious  threat,  to  the  security  of  the 


United  States  of  America.  It’s  the  Pen¬ 
tagon  bureaucracy”  (Rumsfeld,  2001). 
The  unfortunate  events  of  the  next  day 
undoubtedly  lessened  the  impact  of 
these  words  on  the  change  process. 
However,  they  still  are  evidence  of  the 
intent,  vision,  and  determination  to 
transform  the  acquisition  system. 

Rumsfeld’s  vision  contained  several 
key  elements.  Notably  he  again  empha¬ 
sized  the  concept  of  commercial  out¬ 
sourcing  to  save  money  and  a  renewed 
emphasis  on  doing 
only  the  functions  di¬ 
rectly  related  to 
warfighting.  With  re¬ 
spect  to  technology, 
he  called  for  new  ef-  ? 
forts  to  streamline  the 
development  process 
to  catch  up  with  pri¬ 
vate  sector  develop¬ 
ment  cycles.  Finally,  he  made  a  strong 
case  for  improving  the  retention  of  a 
quality  workforce  in  the  entire  military 
from  the  uniformed  personnel  to  the  ac¬ 
quisition  corps.  This  last  goal  was  prob¬ 
ably  the  most  significant  new  focus  and 
reflected  a  growing  realization  during 
the  late  nineties  that  the  DoD  was  los¬ 
ing  ground  on  the  labor  front. 

The  Goldwater  Nichols  Act  of  1986 
coupled  with  the  Secretary  of  Defense 
Dick  Cheney’s  Defense  Management 
Review  of  1989,  formed  the  groundwork 
for  a  professional  acquisition  workforce 
by  “establishing  in  each  military  depart¬ 
ment  a  dedicated  corps  of  military  officers 
who  will  be  acquisition  specialists”  (DoD 
DMR,  1989).  This  sweeping  change  was 
not  accepted  across  the  board  as  a  good 
thing  for  the  services,  but  according  to 
the  General  Accounting  Office  (GAO) 


"The  events  of 
September  11, 
2001  raised 
dramatically 
the  urgency  of 
solving  acquisition 
problems." 
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investigations  conducted  after  1986,  the 
Army  was  clearly  leading  the  way  to  im¬ 
plementing  the  provisions  of  Goldwater- 
Nichols  and  Defense  Management  Re¬ 
view  (DMR),  Major  General  Dick 
Stephenson,  then  the  senior  Army  avia¬ 
tion  acquisition  professional  on  active 
duty,  stated  that  the  full  implementation 
of  the  Army  Acquisition  Corps  would 
result  in  the  “formation  of  another  Army 
Veterinary  Corps”,,,  where  the  officers 
of  the  corps  would  lose  all  credibility  with 
the  warfighting  side  of  the  Anny, 

On  the  civilian  side  of  the  labor  force, 
as  a  result  of  successive  hiring  freezes  and 
senior  grade  restrictions,  the  average  age 
of  the  workforce  is  rapidly  approaching 
50  years  of  age  with  few  experienced 
acquisition  civilians  ready  to  fill  the  gap  of 
the  retiring  workforce. 
Over  the  last  ten  years, 
the  acquisition  workforce 
grew  farther  and  farther 
apart  from  the  main 
stream  Army.  This  rift 
in  cultures  between  the 
warfighters  and  acquisi¬ 
tion  officers  forced  the 
most  senior  officers  in 
the  Army  to  question  the 
net  value  of  acquisition  general  officers 
to  the  business  case  of  the  Army. 

The  Comanche  program,  since  its 
inception  as  the  Light  Helicopter  Experi¬ 
mental  (LHX),  was  managed  by  a  Gen¬ 
eral  Officer;  but  in  2001,  with  the  Coman¬ 
che  budget  exceeding  $1  billion  annually, 
the  Army  downgraded  the  Program  Man¬ 
ager  (PM)  position  to  a  Colonel.  Ironi¬ 
cally,  this  degradation  of  rank  compared 
to  authority  and  responsibility  is  in  con¬ 
trast  to  the  warfighting  changes  to  the 
Army  in  the  Objective  Force,  In  which 


more  senior  ranking  personnel  command 
smaller  units.  Between  2001  and  2002, 
the  Comanche  Program  Management 
Office  (PMO)  lost  over  120  years  of  ci¬ 
vilian  experience  as  all  of  the  most  senior 
acquisition  personnel  in  the  PMO  attained 
retirement  age  and  departed  government 
service  over  a  nine-month  period.  In  or¬ 
der  to  fill  these  vacancies,  the  PM  was 
forced  to  seek  candidates  outside  of  civil 
service  in  order  to  find  qualified 
replacements  because  no  qualified  gov¬ 
ernment  candidates  applied.  This  trend 
seems  to  support  the  shift  in  vision  by 
Rumsfeld  toward  development  of  the  ac¬ 
quisition  corps  Itself, 

Cancellation  of  5000  Series  — 

October  30, 2002  USD  Paul  WoLrownz 

On  August  29,  2002,  a  draft  memo 
was  circulated  from  the  Secretary  of 
Defense  titled:  Operation  of  the  Defense 
Acquisition  System  issuing  interim  guid¬ 
ance  for  the  acquisition  community  in 
light  of  the  pending  cancellation  of  the 
5000  series  of  documents.  The  memo  to 
cancel  the  5000.1  D  (directive),  the 
5000,2  I  (instruction),  and  the  5000.2  R 
(regulation)  documents  was  later  issued 
by  the  DUSD  (AT&L)  Paul  Wolfowitz 
on  October  30,  2002.  Canceling  all  of 
these  sent  a  strong  signal  throughout  the 
acquisition  corps  that  incremental  and 
piecemeal  programmatic  approaches 
were  not  acceptable  solutions  to  the 
transformation  problem. 

The  cancellation  memo  laid  out  a  clear 
message  by  stating  that  the  5000  docu¬ 
ments  are  “overly  prescriptive  and  do  not 
constitute  an  acquisition  policy  environ¬ 
ment  that  fosters  efficiency,  creativity,  and 
innovation.”  It  went  on  to  state  further  that 
the  interim  guidance  issued  separately  is 


"Over  the  lust 
fen  years,  the 
acquisition  work 
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apart  from  the 
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Table  1 .  The  Seven  Key  Documents  Reviewed 


Document 

Title 

Definition  of  Problem 

Acquisition  Impact 

Focus  of  Effort 

National 

Performance 

Review 

VP  Al  Gore 
Sept.  7, 1993 

From  Red  Tape  to 
Results:  Creating 
a  Government 
that  Works  Better 
and  Costs  Less 

1.  Rules  made  for  era  of 
stratified  retailing. 

2.  Overly  focused  on 
fraud  prevention. 

3.  Govt  required  specs  on 
non-essential  items. 

4.  Govt  spends  more  for 
same  commercially 
available  parts. 

5.  Private  sector 
compliance  burden  of 
regulations. 

6.  Proliferation  of 
unneeded  regulations. 

1 .  Unnecessary  rules  and 
procedures. 

2.  Heavy  burden  of 
bureaucracy  and 
procedures. 

3.  Adds  time  to  process 
and  paperwork. 

4.  Wastes  money  on  spec¬ 
ifications  not  important. 

5.  Compliance  cost 
estimated  at  $430 
Billion/year. 

6.  Lack  of  change 
motivation  (system 
inertia). 

1.  Simplify  the 
Procurement 
Process. 

2.  Rely  More  on 
Commercial 
Marketplace. 

3.  Eliminate  the 
Regulatory 
Burden. 

Perry  Memo 
Plan 

Delivered  to 
House  Armed 
Services 
Committee 
and 

Governmen¬ 
tal  Affairs 
Committee 
Feb.  9,  1994 

Acquisition 

Reform:  A 

Mandate  for 
Change 

Maintaining 
Technological 
Superiority  &  a 
Strong  National 
Industrial  Base 

1.  Industrial  Age  Big 
Contracts  Mentality. 

2.  Low  Risk  and  High 
Control  of  Process. 

3.  DoD  small  %  of 
commercial  purchases. 

4.  Cycle  Time  10+ years 
vs  industry  4  yrs. 

5.  Compliance  burden  of 
regulations. 

6.  Dispersed  functional 
responsibility. 

1 .  Consolidation  of  indus-try 
and  erosion  of  base. 

2.  Obsession  with  system, 
process  and  programs. 

3.  Lack  of  interest  from 
commercial  sector  (no 
wins). 

4.  Deployment  of  obsolete 
technology. 

5.  Unnecessary  cost, 
wastes  money  for  field. 

6.  Lack  of  change 
motivation  (system 
inertia). 

1.  Cultural 

Change. 

2.  Technology 
Flow. 

3.  Bureaucracy. 

DUSD(AR)  ! 
Acquisition 
Reform  - 
Colleen 
Preston  - 
1995 

Reengineer 

the 

Acquisition 

System 

Statement  by 
DUSD  (AR)  Mrs. 
Colleen  Preston 
on  Acquisition 
Reform  before 
Committee  on 
National  Security 
US  House  of  Rep. 

1.  Warfighter  is  Customer 
but  is  not  present. 

2.  Change  is  not  em¬ 
braced  within  the  DoD. 

3.  Do  not  emulate 
commercial  best 
practices. 

4.  Too  many  defense  only 
suppliers. 

5.  Not  oriented  on  perfor¬ 
mance  outcomes. 

1.  Long  Acq  Lead  times 

2.  No  one  willing  to  take 
any  risk. 

3.  High  costs  and 
inefficient. 

4.  Some  suppliers  are 
not  willing  to  sell  to 

DoD. 

5.  Excessive  protests 
filed  as  way  of  getting 
business. 

1.  Improve 
Responsive¬ 
ness. 

2.  Reduce  Costs. 

3.  Facilitate 
Merger  of 
Defense  and 
Commercial 
Industrial 

Bases. 

Defense 
Reform 
Initiative 
Report 
(DRI)  1997 

William 
Cohen 
Secretary  of 
Defense 

Leading  Change 
in  a  New  Era 

(The  DR!  Office 
was  officially 
launched  in  May 
1998) 

1.  Lack  of  interest  from 
commercial  sector. 

2.  Consolidation  of 
industry;  lack  of  base. 

3.  Revolution  in  Military 
Affairs  needs  support 
to  work. 

1.  Potential  deployment 

of  obsolete  technology. 

2.  Obsession  with  system, 
lack  of  motivation  to 
change. 

3.  Unnecessary  costs 
wastes  money  for  field. 

1.  Reengineer. 

2.  Consolidate. 

3.  Compete. 

4.  Eliminate. 
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Table  1.  The  Seven  Key  Documents  Reviewed  (continued) 


Document 

Title 

Definition  of  Problem 

Acquisition  Impact 

Focus  of  Effort 

The  Road  Ahead 
June  2, 2000, 

USDfAT&L) 

J.S,  Gansler 

Launch  of  the 
Acquisition 
Initiatives  Office 
in  the  OSD 

Accelerating  the 
Transformation  of 
Department  of 
Defense 
Acquisition  and 
Logistics 
Processes  and 
Practices 

1.  Supporting  the  RMA 
to  meet  threats  of 
asymmetric  warfare. 

2.  Need  for  high 
interoperability 
among  U.S.  Forces, 

3.  Development  dollars 
squeezed  by  flat 
budget  lines  and 
rising  support  costs. 

1.  Acquisition  support 
system  not  in  tune  with 
RMA  (needs  RBA  in 
department). 

2.  Too  many  barriers  to 
change. 

3.  Not  keeping  pace  with 
technology  develop¬ 
ment  (falling  R&D 
investments  and  cycle 
times). 

1.  Reduce 
acquisition 
cycle  times  for 
technology. 

2.  Lower  total 
ownership 
costs, 

3.  Reduce 
overhead  of 
support  (A&L). 

Rumsfeld’s 

Vision  of 

Reform 

Sept,  10, 2001 

Bureaucracy  to 
Battlefield 

1.  Waste  of  resources. 

2.  Outdated  systems. 

3.  Redundant 
processes. 

1.  Supporting  the  RMA  to 
meet  threats  of 
asymmetric  warfare. 

2.  Need  for  high  inter¬ 
operability  among  U.S. 
Forces. 

3.  Development  dollars 
squeezed  by  flat 
budget  lines  and  rising 
support  costs. 

1.  Reduce  cycle 
time. 

2.  Improve 
workforce 
morale. 

3.  Strengthen 
health  of 
industrial  base, 

4.  Leverage 
commercial 
technology 
insertion. 

Cancellation  of 
the  5000  Series 
Wolfowitz,  2002 

Rapidly  deliver 
affordable, 
sustainable 
capability  to 
meet  warfighter 
needs 

Draft  Memo 
Circulated 
(later  signed) 

1.  Current  5000  set  is 
overly  prescriptive 
(The  PM  shall...). 

2.  5000  does  not 
create  the  desired 
new  acquisition 
policy  environment. 

3.  Does  not  foster 
efficiency,  creativity, 
and  innovation. 

1.  Remove  proscriptive 
nature  of  5000  series  to 
allow  more  PM 
discretion. 

1.  Flexible. 

2.  Innovative, 

3.  Speed  in 
technology 
deployment. 

to  “rapidly  deliver  affordable,  sustainable 
capability  to  the  warfighter  that  meets  the 
warfighter’s  needs  ”  Thus  the  three  tenets 
from  ‘The  Road  Ahead’  of  faster,  cheaper, 
cheaper  were  reiterated  here  in  the  interim 
guidance  but  the  driving  reason  for 
change  was  still  a  need  for  fundamental 
cultural  change  (fostering  the  favorable 
policy  environment). 

The  faster/cheaper  message  does  not 
carry  the  same  kind  of  urgency  for  deep 
change  that  the  Perry  memo  called  for. 


Thus,  it  is  understandable  that  the  DRI 
and  its  successor,  the  OSD  AI  office,  op¬ 
erated  primarily  from  a  mandate  of  pro¬ 
grammatic  fixes  to  demonstrate  cost  sav¬ 
ings/avoidance.  A  year  after  the  Rumsfeld 
September  10,  2001  call  for  an  overhaul 
to  the  Pentagon  bureaucracy,  the  efforts 
(5000  updates,  program  management 
tools  implementation,  etc.)  were  still  not 
structured  in  a  way  that  could  add  up  to 
the  fundamental  type  of  change  de¬ 
manded  by  Perry  in  1994,  Gansler  in 
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1999,  or  Rumsfeld  in  2001.  Table  1  sum¬ 
marizes  the  key  points  from  each  of  the 
eight  documents  reviewed  and  forms  the 
baseline  of  reference  for  this  paper. 

Next  we  examine  the  eight  documents 
in  light  of  the  key  drivers  of  acquisition 
reform.  Then  we  return  to  look  at  the  ten¬ 
sions  remaining  in  the  change  vision  and 
what  can  be  learned  from  this  decade  of 
change  effort. 

The  Need  for  Change 


Changing  Needs  of  Warfighters 

Many  recurring  issues  have  driven 
efforts  at  change  in  the  DoD  over  the  last 
decade.  Three  key  drivers  continue  to  sur¬ 
face  throughout  the  statements  and  docu¬ 
ments  relevant  to  transformation.  First  are 
the  changing  demands  of  the  warfighters. 
The  battlefield  has  become  a  technologi¬ 
cal  platform.  To  respond  to  growing  un¬ 
conventional  threats  and  asymmetrical 
warfare,  military  requirements  have  been 
shifting  over  the  last  decades. 

For  example,  the  Comanche  program 
has  evolved  over  the  years  to  meet  a  new 
set  of  requirements  and  warfighter  needs. 
The  Comanche  (RAH-66)  is  now  more 
of  a  systems  integrator  to  the  overall  force 
than  a  modern  attack  helicopter.  By 
morphing  its  role  it  has  remained  relevant 
to  the  new  needs  of  the  force.  At  the  same 
time,  the  program  has  languished  for 
decades  competing  for  funds  and  atten¬ 
tion  to  actually  bring  the  technology  to 
the  battlefield.  It  has  faced  serious  tech¬ 
nological  obsolescence  issues  due  to  long 
development  cycles  and  less  than  desir¬ 
able  program  structuring  (Birmingham, 
2002). 


The  vision  of  transformation  as  driven 
by  the  changing  needs  of  warfighters 
(customers)  is  outlined  in  Table  2.  The 
term  warfighter  does  not  appear  in  the 
NPR,  but  it  was  quickly  adopted  within 
the  DoD  to  mean  the  ultimate  customer 
of  the  acquisition  sys¬ 
tem.  Since  the  needs  of 
the  customer  were 
changing,  the  support 
structure  of  the  acqui¬ 
sition  community  must 
change  to  meet  those 
evolving  customer 
needs.  The  warfighter 
as  customer  seemed  to 
lose  some  focus  near 
the  end  of  the  1990s 
not  appearing  in  docu¬ 
ments  relating  to  the 
DRI  and  OSD  AI. 
progresses,  more  and  more  attention  was 
given  to  the  technology  flows,  the  shift¬ 
ing  defense  industrial  base,  and  slipping 
market  power  of  the  DoD  in  breaking 
technological  fields  perhaps  assuming  the 
warfighter  needs  were  being  addressed. 

Shifts  in  Military-Industry 
Technology  Flows 

The  second  driver  closely  linked  to 
the  first  is  the  commercial  rate  of  tech¬ 
nological  development.  Military  acqui¬ 
sition  cycles  are  as  much  as  2.5  times 
longer  than  commercial  cycles.  Techno¬ 
logical  advances  can  quickly  appear  on 
the  threat  horizon  creating  the  very  real 
possibility  that  U.S.  Military  personnel 
could  be  technologically  inferior  to  a 
foe  in  the  field  solely  due  to  the  slow¬ 
ness  of  the  acquisition  support  process. 
Secretary  Rumsfeld  directly  attacked 


'The  term 
warfighter 
does  not  appear 
in  the  NPR,  but 

I  it  was  quickly 
adopted  within 
the  DoD  to  mean 
the  ultimate 
customer  of 
the  acquisition 
system." 
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As  the  decade 


49 


Defense  Acquisition  Review  Journal — January-April  2004 


Table  2.  Tracing  the  Changing  Needs  of  the  Customer  (Warfighter) 


Reference  Document 

Characterization  of 

Acquisition  Problem 

Reference  to  Warfighters  Needs 

National  Performance  Review 
Gore,  1993 

Industrial-Era  Bureaucracies  in  an 
Information  Age” 

“tough  global  competition.” 

“demanding  customers.” 

Perry  Memo,  1994 

Current  acquisition  system  does  not 
support  a  strong  national  defense 
industrial  base. 

‘the  threats  are  changing  and 
unpredictable.” 

Colleen  Preston,  1995  (p.  2) 

“new  national  security  challenges 
require  a  more  flexible,  agile  and 
timely  acquisition  process.” 

“meet  warfighter  needs”...  we  must 
be  ‘capable  of  meeting  unpredict¬ 
able  needs.* 

Lead  times  are  too  long  to  field 
equipment/technology. 

Warfighter  is  customer  but  is  not 
present  in  the  process. 

Defense  Reform  Initiative,  1997 

Reallocating  resources  ‘from  overhead 
and  support  activities  to  our  fighting 
forces. 

Enhancing  efforts  to  defend 
against  asymmetric  threats. 

Joint  Vision  2010;  information  use 
and  denial  in  battle. 

The  Road  Ahead,  1999 

Revolution  in  Military  Affairs  must  be 
supported  by  Revolution  in  Business 
Affairs. 

Unpredictable  threats,  rogue 
nations,  use  of  weapons  of  mass 
destruction  (WMD). 

Rumsfeld’s  Vision,  2001 

Pentagon  Bureaucracy. 

Institutional  Inertia. 

Must  change  to  meet  new  world 
challenges  of  multiple  threats  of 
unpredictable  source  and  nature. 

Dollar  wasted  is  one  denied  to  the 
warfighter  (efficiency)  Overcapacity  of 
bases. 

Excellence  in  functions  related  to 
warfighting. 

Wolfowitz’s  Memo 

Cancelling  the  5000  Series 

Overly  prescriptive; 

Does  not  foster  efficiency,  creativity 
and  innovation. 

“to  rapidly  deliver  affordable, 
sustainable  capability  to  the 
warfighter  that  meets  the 
warfighter’s  needs,” 

this  bureaucratic  threat  on  September 
10?  2001  when  he  outlined  his  vision 
for  the  new  way  of  doing  business  in 
the  Pentagon.  One  outcome  of  changes 
in  technology  flows  is  that  the  military 
has  found  itself  struggling  to  keep 
pace  with  technological  developments 
due  to  either  disinterest  on  the  part  of 


commercial  developers  or  cumbersome 
DoD  acquisition  systems. 

The  dynamics  behind  this  shift  in 
market  power  (the  DoD  used  to  be  the 
major  buyer  of  technology)  goes  back  to 
the  aftermath  of  World  War  IL  As  war¬ 
time  production  ended,  the  Cold  War 
helped  sustain  predictable  and  steady 
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Table  3. 


Tracing  the  Changing  Nature  off  Technology  and  Industry  Dynamics 


Reference  Document 

Characterization  of 

Acquisition  Problem 

Reference  to  Industry  and 
Technology  Dynamics 

National  Performance  Review 

Gore,  1993 

“Industrial-Era  Bureaucracies  in  an 
Information  Age” 

Faulty  economic  assumptions  of 
scale  efficiency  and  faulty 
managerial  assumptions  of  audit, 
compliance  and  control. 

“tough  global  competition” 

“demanding  customers” 

Perry  Memo,  1994 

“shrinking  defense  industrial  base;” 

"commercial  technology  advance¬ 
ments  are  outpacing  DoD 
sponsored  efforts.” 

1965  Military  consumed  75%  of 
semiconductor  production;  1995 
consumed  around  1%. 

Commercial  design  cycle  is  3-4  yrs; 
DoD8-10yrs. 

Acquisition  Reform,  DUSD(AR) 
Colleen  Preston,  1995 

Too  many  defense-only  suppliers. 

“facilitate  the  merger  of  the  defense 
and  industrial  bases. 

Some  suppliers  unwilling  to  sell  to 

DoD  (hassle  factor  prohibitive). 

Not  emulating  best  commercial 
practices. 

Defense  Reform  Initiative, 

1997 

Consolidation  of  industry  and 
erosion  of  core  capabilities. 

RMA  will  outrun  ability  of  Acq 

System  to  support  it. 

Lack  of  interest  from  commercial 
sector. 

Deployment  of  outdated  technology 
to  the  field. 

The  Road  Ahead,  1999 

Logistics  response  too  slow; 

Acq  cycle  too  long. 

Integrate  a  civil-military  industrial 
base. 

Rumsfeld’s  Challenge,  2001 

Bureaucratic  inertia; 

Excess  infrastructure. 

PPBS  outdated. 

Technology  moves  faster  than  the 

DoD  System. 

Deploying  outdated  technology. 

Cancellation  of  5000 

Series,  Wolfowitz,  2002 

Rapid  movement  from  S&T  to 
Deployment  and  Fieldinglntegrated 
T&E  Improved  Cycle  Time 

Overly  prescriptive  regulations  do 
not  foster  innovation  and  flexibility 
in  program  management  to  keep 
up  with  technology. 

TERMS: 

PPBS  -  Program  Planning  Budget  System 

S&T  -  Science  and  Technology 

T&E  -  Test  and  Evaluation 

RMA  -  Revolution  in  Military  Affairs 
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improvements  in  materiel  and  systems  to 
counter  the  static  Soviet  threat.  As  the  tech¬ 
nology  boom  of  the  nineties  took  off  and 
the  Cold  War  ended,  industry  rushed  off 
to  richer  commercial  markets  and  the 
military  industrial  base  was  threatened 
with  an  erosion  of  core  capability.  The 
defense  industry  consolidated  as  major 
program  awards  dwindled  and  the  predict¬ 
ability  of  defense  business  declined  rapidly. 
As  has  happened  in  other  consolidating 
industries  —  railroads,  steel,  and  telecom¬ 
munications  —  the  power  of  unions  be¬ 
came  concentrated  limiting  further  the 
ability  of  defense  contractors  to  control 
labor  costs  or  move  production  work  to 
more  efficient  locations. 

In  tracing  the  vision  of  acquisition 
reform  as  it  relates  to  technology,  two 
themes  are  intertwined:  loss  of  DoD 
market  influence  on  the  defense  industry 
and  cycle  time  gap  between  military  and 
commercial  technology  applications.  The 
vision  from  Gore  and  Perry  focused  on 
dealing  with  shrinking  military-industrial 
base  by  first  commercializing  the  tech¬ 
nology  developed  for  the  military  thereby 
helping  industry  to  profit  from  defense 
related  work;  and  second,  by  using  more 
commercial  technology  in  defense 
programs  thereby  lowering  the  DoD 
acquisition  costs. 

The  twin  needs  of  speed  and  support 
are  embedded  in  these  efforts  but  seem 
to  gain  more  focus  toward  the  end  of  the 
decade.  Notice  in  Table  3  how  the  health 
of  the  defense  industry  is  an  objective  up 
through  1995.  It  still  receives  mention  in 
1997  and  2001  but  more  as  a  principle  of 
change,  not  a  focus  of  change  itself.  Thus, 
the  vision  for  change  with  respect  to 
technology  seems  to  have  shifted  from 
building  a  new  military  industrial  reality 


to  more  simply  reducing  technology 
deployment  cycle  time. 

Another  way  to  interpret  this  is  that  the 
focus  shifted  from  a  perceived  complex 
root  cause  to  a  tangible  demanding  re¬ 
sult,  It  is  also  possible  that  the  vision 
shifted  because  the  threat  of  technologi¬ 
cal  inferiority  became  a  real  crisis.  Thus, 
the  industry  alignment  became  eclipsed 
by  the  immediate  need  to  get  technology 
to  the  troops  faster.  The  shift  makes  it 
easier  for  Rumsfeld  to  connect  change 
(meaning  rapid  fielding  of  technology) 
to  meeting  warfighter  needs.  It  is  prob¬ 
ably  much  more  difficult  to  take  on  fix¬ 
ing  the  defense  industry  capability,  which 
is  fraught  with  political,  labor,  economic, 
and  international  concerns  that  make  it 
difficult  to  see  meaningful  progress. 

Table  3  outlines  the  key  elements  of 
the  change  vision  as  it  relates  to  technol¬ 
ogy  throughout  the  decade. 

Defense  Budget  Constraints 

The  third  key  driver  of  change  is  bud¬ 
get  and  spending  constraints.  The  initial 
vision  declared  that  the  government  must 
actually  spend  less  and,  indeed,  defense 
budgets  had  been  declining  in  the  late 
eighties  and  through  most  of  the  nineties. 
About  half  way  through  the  decade  the 
emphasis  seems  to  shift  from  an  absolute 
cost  reduction  to  a  reduction  in  overhead 
allowing  more  money  for  technology  and 
R&D.  This  focus  is  most  notable  in 
Rumsfeld’s  comments  on  September  10, 
2001  when  he  spells  out  a  vision  for  not 
reducing  overall  cost  but  for  reducing 
waste  allowing  for  more  productive  use 
of  the  same  funding.  This  part  of  the 
vision  probably  changed  the  most  and 
even  more  so  after  September  11,  2001. 
As  defense  budgets  began  to  expand 
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Table  4.  Tracing  the  Shrinking  Defense  Budget  as  a  Driver  of  Change 


Reference  Document 

Characterization  of  Acquisition 

Problem. 

Reference  to  Changing  Defense 
Budgets. 

National  Performance  Review 

Gore,  1993 

‘The  federal  government  seems 
unable  to  abandon  the  obsolete.” 

“Waste  and  inefficiency;  loss  of  faith 
of  taxpayer  to  fund  it.” 

“government  must  cost  less” 

“reducing  the  federal  deficit” 

Perry  Memo,  1994 

DoD  pays  more  for  same  parts  due  to 
regulatory  burden. 

Lack  of  access  to  commercial  supply 
and  surge  production. 

Defense  spending  in  real  terms 
has  declined  by  40%  from  FY 

1985  to  FY  1 997. 

Acquisition  Reform  DUSD(AR) 

Colleen  Preston,  1995 

More  firms  become  defense  only 
suppliers  (dependent). 

Reduce  costs; 

Procure  best  value  goods  and 
services. 

Defense  Reform  Initiative,  1997 

Revolution  in  Business  Affairs  must 
support  Revolution  in  Military  Affairs 

Resources  must  be  freed  up  to 
invest  in  new  R&D. 

The  Road  Ahead,  1999 

Total  ownership  costs  too  high. 

Support  costs  and  overhead  growing 
too  fast. 

Reduce  support  costs; 

Reduce  overhead  costs. 

Rumsfeld’s  Challenge,  2001 

Excess  infrastructure  (bases). 

Redundant  staff  and  agencies. 

Inefficient  systems/processes. 

Public  trust  for  tax  dollars  spent 
on  defense; 

Waste  drains  resources  needed 
for  addressing  new  threats. 

Cancellation  of  5000  Series, 

Wolfowitz,  2002 

Need  total  systems  approach  to 
acquisition  management. 

Performance  based  logistics 

Cross  service  acquisition 
agreements 

Overly  prescriptive  regulations 
do  not  foster  efficiency  of 
operations;  cost  realism  or 
program  stability. 

again,  the  focus  shifted  even  more  to 
spending  it  on  the  right  things  and  par¬ 
ticularly  getting  the  technology  into  the 
warfighters  hands  faster  and  more  effi¬ 
ciently.  Figure  1  shows  overall  defense 
spending  during  the  decade  of  transfor¬ 
mation. 


Table  4  traces  the  vision  of  change 
as  driven  by  budgetary  concerns.  The 
shift  in  perspective  roughly  correlates 
with  the  change  in  administrations,  the 
end  of  Base  Realignment  and  Closure 
(BRAC)  (93  &  95)  rounds  and  the  on¬ 
set  of  the  terrorist  war.  One  thing  is  clear 
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Information  Technology  Projects  ACAT IA  -  III 

ACAT  IA 

ACAT  II 

ACAT  ill 

Total  Life  Cycle  Costs 

>  $378, 000, MO 

N/A 

<  ACAT  1 

Total  Program  Cost 

>  $126,000,000 

m 

<  ACAT  1 

Total  Program  Costs  (all  appropria¬ 
tions)  in  any  single  year 

>  $32,000,000 

N/A 

<  ACAT  1 

Note:  All  $  amounts  are  expressed  in  FY2000  constant  dollars 
Source:  DOD  Instruction  5000.2,  May  12,  2003,  Enclosure  2 


Figure  1*  Percent  Real  Growth  in  Defense  Spending 


—  the  spending  patterns  will  not  be  the 
same  as  in  the  post-World  War  II  and 
Cold  War  periods.  Large  multi-year  cost- 
plus  contracts  are  gone.  The  new  envi¬ 
ronment  of  performance-based  contracts 
and  milestone  development  protects  the 
budget  from  being  devoured  by  program 
growth  but  at  the  same  time  has  intro¬ 
duced  a  high  level  of  uncertainty  into  the 
program  management  side  of  acquisition. 
This  has  in  turn  affected  the  relationship 
between  the  DoD  and  industry  exacer¬ 
bating  the  already  weak  industrial  base. 
It  is  also  likely  that  continued  budgetary 
constraints  because  of  the  demands  of 
homeland  security  will  make  cost  effi¬ 
ciency  and  cost-effectiveness  essential  to 
all  military  spending  plans. 

Observations  for 
Further  Empirical  Research 


Lessons  Learned  from  the  Decade 

What  does  the  review  of  these  docu¬ 
ments  tell  us?  There  are  several  general 
observations  and  many  research  questions 


generated  from  a  review  such  as  this.  First, 
visions  change  with  vision  owners  and 
changing  visions  make  it  difficult  to  main¬ 
tain  change  momentum.  Changing  vi¬ 
sions  also  create  potential  for  less  than 
full  realization  of  change  consequences 
as  change  agents  become  overly  focused 
on  achieving  some  measure  of  change 
during  their  term  in  power.  This  empha¬ 
sis  on  demonstrable  action  and  results  can 
diminish  the  effectiveness  of  even  well 
thought  out  visions  and  plans. 

Second,  visions  do  naturally  evolve 
with  time.  Not  just  because  of  changing 
vision  owners  but  also  because  of  chang¬ 
ing  environmental  conditions  as  with  the 
changing  threats  the  U.S.  Military  must 
counter.  But  changing  environments  also 
come  from  within  as  in  the  changing 
workforce,  federal  labor  acts,  and  tech¬ 
nology  driven  product  life  cycles.  In  the 
midst  of  change  —  of  the  change  vision 
itself  —  clear  pictures  of  the  desired  end 
states  may  be  even  more  important.  Thus, 
it  would  seem  important  for  leaders  to 
focus  on  maintaining  a  clear  and  com¬ 
pelling  vision  (picture)  of  where  we  are 
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going  in  the  midst  of  evolving  change 
plans  and  programs. 

Tensions  in  the  Change  Process 

Faster  vs*.  Cheaper 

In  examining  the  status  and  number 
of  DoD  program  starts,  restarts,  and  can¬ 
cellations  over  the  last  ten  years,  it  is  ob¬ 
vious  that  the  DoD  track  record  for  keep¬ 
ing  programs  on  schedule  and  within  cost 
has  not  been  impressive.  Both  industry 
and  DoD  program  manager’s  have  suf¬ 
fered  from  a  contagious  trend  of  unmer¬ 
ited  optimism  in  defining  and  support¬ 
ing  both  cost  and  schedule  program  risks, 
especially  across  the  most  complex  pro¬ 
grams  such  as  V-22,  F-22,  and 
Comanche.  The  initial  program  baselines 
were  built  around  making  the  programs 
fit  inside  a  constricting  cost  and  sched¬ 
ule  box  vs.  designing  program  plans 
within  flexible  boxes  to  accommodate 
the  many  unknowns  associated  with  com¬ 
plex  integration  initiatives.  The  current 
DoD  acquisition  administration’s  recog¬ 
nition  of  the  problem  has  been  outlined 
with  recent  Cost  Analysis  Integration 
Group  (CAIG)  guidance  to  put  realism 
into  the  program  plans.  Evidence  of  this 
realism  can  be  seen  in  the  high  number 
of  program  Nunn-McCurdy  breaches 
occurring  over  the  past  few  years. 

In  Army  aviation  alone,  Comanche, 
Chinook,  and  Blackhawk  have  all  ex¬ 
ceeded  program  breach  thresholds  from 
baselines  between  2001  and  2003,  base¬ 
lines  that  were  inadequately  established 
for  reasons  of  keeping  the  programs  alive 
and  on  schedule.  Further  examination  of 
the  Comanche  history  indicates  a  perva¬ 
sive  give  it  ALL  to  me  faster  philosophy 
despite  changes  to  requirements  and  the 
reality  that  not  all  capabilities  can  be 


delivered  within  the  same  program  risk 
profile.  Despite  efforts  from  OSD  AT&L 
leadership  to  make  the  TRADOC  com¬ 
munity  accountable  for  the  approved  set 
of  requirements  in  addition  to  require¬ 
ments  creep,  few  in  TRADOC  recognize 
or  understand  programmatic  impacts  to 
Cost  As  an  Independent  Variable  (CAIV) 
analyses. 

General  Abrams,  in  April  2001,  in  a 
briefing  to  the  Army  Chief  of  Staff  in 
which  he  was  justifying  a  new  set  of 
Objective  Force  requirements  for  the  pro¬ 
gram  plan,  stated  that  the  “Comanche 
weight  growth  experi¬ 
enced  over  the  last  13 
years  is  the  result  of  the 
Army  Acquisition  Corps’ 
inability  to  keep  discipline 
within  the  requirements 
process.”  Ironically,  this 
was  the  same  briefing  in 
which  the  Program  Man¬ 
ager  stated  to  the  Chief  of  Staff,  Army 
(CSA)  that  based  on  the  current  contrac¬ 
tor  performance  and  expanded  set  of  re¬ 
quirements,  “all  program  goals  and  ob¬ 
jective  could  not  be  met.” 

The  complexity  of  the  interrelation¬ 
ship  between  TRADOC  and  Acquisition 
is  further  strained  as  DoD  programs  con¬ 
tinuously  undergo  a  systemic  decrement 
of  program  funding  after  contracts  are 
signed  and  executed.  The  prospect  to  do 
smart  things  cheaper  and  faster  is  a  dual 
edge  sword.  Development  Program  Man¬ 
agers  on  both  the  industry  and  DoD  side 
must  constantly  search  for  ways  to  do 
business  cheaper  and  faster... just  to  keep 
up  with  the  annual  drain  of  funding. 

For  fielded  system  Program  Manag¬ 
ers,  the  problem  of  doing  business 
cheaper  and  faster  is  often  confused 


"The  prospect 
to  do  smart 
things  cheaper 
and  faster  is 
a  dual  edge 
sword." 
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between  efforts,  which  are  cost  saving  vs. 
cost  avoiding.  Cost  saving  efforts,  almost 
always  will  be  accompanied  with  a  re¬ 
duction  in  program  budgets  far  ahead  of 
the  realization  of  the  cost  savings.  Cost 
avoidance  is  a  more  agreeable  term  for 
the  near  term  but  pays  the  same  penalty 
in  the  out-years. 

Standardized  vs.  Latest  Technology 
The  concept  of  evolutionary  acquisi¬ 
tion  is  not  new.  For  the  past  ten  plus  years, 
DoD  program  managers 
have  modified  their 
fielded  systems  with 
technology  insertions, 
usually  benefiting  from 
commercial  innova¬ 
tions.  Over  the  last  ten 
years,  the  shift  from  a 
DoD  centric  technology 
base  —  where  much  of 
the  warfighting  technol¬ 
ogy  was  developed  for  and  within  the 
DoD  military-industrial  base  to  —  com¬ 
mercial  centric  technology  development. 
DoD  programs  have  become  hostages  to 
the  velocity  and  pace  of  commercial  stan¬ 
dards  and  demands.  To  survive,  program 
managers  had  to  shift  their  development 
and  technology  insertion  strategies  from 
leader  to  follower  —  often  becoming  the 
tail  end  of  scalable  products.  Although 
this  strategy  has  helped  to  slow  down  the 
pace  of  obsolescence,  little  has  been  done 
to  address  the  Program,  Planning,  Bud¬ 
geting  and  Execution  System  (PPBES)  and 
process  to  pay  for  these  efforts. 

The  Apache  Forward  Looking  Infra¬ 
red  (FLIR)  technology  lagged  behind  the 
commercial  standard,  not  because  of 
availability  of  technology  but  because 
ftmding  was  not  planned  well  in  advance 


"Planning 
far  technology 
upgrades  and 
phased  recapital¬ 
ization  lacks 
support  across 
the  DoD  budget 


of  the  technological  maturity  and  test  com¬ 
plete  dates.  Comanche  has  changed  its 
mission  processing  technology  three 
times  since  1991,  not  because  additional 
processing  capability  was  required  but 
because  INTEL  chose  not  to  manufac¬ 
ture  a  Comanche-only  PI 33/233  and 
transfer  controller.  To  plan  for  a  smooth 
transition  to  production  plan,  the 
Comanche  program  manager  was  chal¬ 
lenged  to  hold  a  development  baseline 
configuration  through  the  first  few  lots  of 
production  in  order  to  keep  development 
on  schedule  and  keep  costs  down.  With¬ 
out  a  special,  congressionally  approved 
reprogramming  action,  the  program  would 
have  been  forced  to  change  processing 
architectures  in  mid-development  — 
blither  delaying  the  fielding  of  the  system. 

Configuration  management  of  DoD 
systems,  particularly  non-commercial  off- 
the-shelf  (COTS)  systems  present  another 
unique  challenge  for  DoD  program  man¬ 
agers.  Since  DoD  systems  are  not  fielded 
like  Ford  automobiles,  multiple  configu¬ 
rations  of  the  same  system  is  a  given . 
Comanche  will  field  6502  aircraft  starting 
in  2008  and  complete  fielding  in  2020. 
Using  the  last  ten  years  as  a  model,  it  is 
likely  that  there  will  be  at  least  four  dif¬ 
ferent  Comanche  configurations  in  our 
Army  when  the  last  lot  of  aircraft  is 
delivered.  Planning  for  technology  up¬ 
grades  and  phased  recapitalization  lacks 
support  across  the  DoD  budget  leader¬ 
ship.  DoD  will  not  plan  for  funding  against 
notional  requirement  changes  and  obso¬ 
lescence  when  current  operations  and 
fiscal  shortfalls  to  current  systems  exist. 
The  consequence  has  been  the  need  for 
greater  Operation  and  Support  (O&S) 
dollars  to  support  multiple  systems,  and 
this  trend  does  not  seem  likely  to  change. 
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Test  management  and  requirements 
has  gained  considerable  momentum  over 
the  last  ten  years  and  has  almost  ignored 
the  rapid  growth  of  models  and  simula¬ 
tions.  Live  fire  testing  requirements  re¬ 
main  a  congressional  mandate  and  DoD 
5000  policy  has  given  the  test  commu¬ 
nity  a  tilted  balance  of  power  in  the  ma¬ 
terial  acquisition  process.  Although  most 
program  managers  support  a  robust  test 
and  evaluation  plan  for  their  systems,  es¬ 
calating  costs  of  redundant  testing  has 
forced  program  managers  to  stretch  pro¬ 
gram  schedules  to  accommodate  re¬ 
quired  operational  test  plans  and  their  as¬ 
sociated  funding  requirements. 

Evolutionary  acquisition  procedures 
have  not  gained  widespread  approval 
from  the  test  community,  as  most  pro¬ 
gram  plans  still  require  a  major,  expen¬ 
sive  graduation  exercise  called  Initial 
Operational  Test  and  Evaluation 
(IOT&E).  Since  the  Low  Rate  Initial  Pro¬ 
duction  (LRIP)  decision  occurs  well 
ahead  of  the  IOT&E,  the  Limited  User 
Test  (LUT)  baseline  for  the  LRIP  Defense 
Acquisition  Board  (DAB)  has  gained  im¬ 
portance  to  become  the  program  plan 
center  of  gravity.  Testers  have  translated 
this  operational  assessment  into  as  much 
of  an  IOT&E  as  possible.  Both  Army 
testers  and  Director  of  Operational  Test 
and  Evaluation  (DOTE)  required  that  the 
Comanche  program  conduct  a  side-by- 
side  test  with  the  Kiowa  Warrior  prior  to 
the  LRIP  DAB  —  to  determine  its  over¬ 
all  suitability.  Both  the  Army  Vice  Chief 
of  Staff  and  Army  Acquisitions  Execu¬ 
tive  questioned  this  philosophy  as  being 
“outdated  and  ridiculous  to  use  an  air¬ 
craft  that  is  being  phased  out  and  has 
everything  known  about  it  on  paper.” 


Finally,  it  is  important  to  ask  if  there 
is  any  difference  between  Rumsfeld’s 
challenge  to  the  Pentagon  in  2001  and 
Perry’s  Mandate  in  1994?  Both  call  for 
radical  change  to  the  normal  way  busi¬ 
ness  is  done  at  the  Pentagon.  Does 
Rumsfeld’s  call  imply  that  Perry’s  vision 
has  not  been  implemented  or  is  it  the  nec¬ 
essary  restatement  of  the  urgency  re¬ 
quired  to  maintain  change  momentum? 
How  much  urgency  is  required  to  keep 
a  vision  alive?  What  causes  a  clear  vi¬ 
sion  to  change  course  and  lose  sight  of 
the  original  goal  or  adopt  a  new  focus? 
Perhaps  further  empirical  research  can 
document  the  effects  of  these  shifts  in 
vision  within  the  DoD  reform  experience. 

Conclusions 


Critics  of  reform  often  say  it  is  all  tied 
up  in  politics  with  little  hope  of  change. 
Deborah  Frank  outlined  this  argument 
succinctly  in  an  article  printed  in  the  Ac¬ 
quisition  Review  Quarterly  journal 
(Frank,  1997).  Her  argument,  based  on 
systems  theory,  suggested  that  with  no 
change  in  the  political  process,  there  is 
little  hope  for  real  change  in  the  acquisi¬ 
tion  process.  Acquisition  funds  are  too 
susceptible  to  political  influence  for  other 
than  military  ends.  While  this  is  always 
true  in  the  American  model  that  gives 
ultimate  control  of  the  military  to  elected 
officials,  it  does  not  preclude  the  ability 
of  a  bureaucracy  to  change  itself  over 
time  driven  by  clear  change  visions. 

We  think  there  are  several  lessons  to 
be  learned  from  this  paper  and  hopefully 
much  more  detailed  research  to  be  done 
into  the  effects  of  the  change  efforts 
already  under  way.  This  review  of 
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transformation  efforts  shows  the  length 
of  time  and  level  of  persistence  needed 
to  effect  real  change  in  a  bureaucracy 
as  large  as  the  DoD.  Continued  strong 


leadership  and  clarity  of  purpose  will 
hopefully  bring  about  deep  and 
substantial  progress  in  the  in  the  years 
ahead. 
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Endnotes 


1 .  The  document  was  actually  signed 
by  DUSD  (AT&L)  J.  S.  Gansler  on 
June  2,  2000  but  had  been  widely 
circulated  within  the  acquisition 
community  since  the  middle  of 
1999. 


2.  The  Army  requirement  was  for  819, 
but  the  Defense  Acquisition  Board 
only  approved  650. 
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TUTORIAL 


AN  INITIAL  LOOK  AT  TECHNOLOGY 
AND  INSTITUTIONS  ON  DEFENSE 
INDUSTRY  CONSOLIDATION 

If  Col  John  D.  Driessnatk,  USAF  and 
Mai  David  R.  King,  PH.D.,  USAF 


Conventional  wisdom  holds  that  defense  industry  consolidation  resulted  from 
decreased  defense  spending.  However,  we  maintain  that  understanding  dynamic 
changes  in  key  defense  institutions  helps  provide  a  more  complete  explanation 
for  observed  consolidation.  Specifically,  we  examine  the  interaction  of  evolving 
technology  and  changing  institutions.  Institutions  reviewed  include  procurement 
policies,  the  weapons  requirements  process,  and  procurement  organizations. 

We  take  an  initial  look  at  the  industry,  and  highlight  how  these  changes 
influenced  transaction  costs  in  the  defense  industry,  more  fully  explain  the  forces 
driving  consolidation,  and  provide  greater  insight  to  policy  makers  seeking  to 
improve  the  performance  of  the  defense  industry.  Further  research  is  needed 
to  build  a  robust  institutional  framework  of  the  defense  industry  and  the  related 
government  agencies  to  allow  better  policy  prescriptions. 

Still  much  of  the  public  discussion  of  weapons  acquisition  problems 
proceeds  as  if  the  terms  “ competition ”  “price,”  “buying,”  and 
“seller”  had  the  meanings  they  do  in  a  market  system . 

(Peck  &  Scherer,  1962) 


Over  the  past  20  years,  75  plus 
United  States  defense  specialized 
firms/divisions  merged  into  five 
major  defense  firms,  or  prime  contrac¬ 
tors  (Commission  on  the  Future  of  the 
Aerospace  Industry,  2002).  Almost  uni¬ 
versally  the  consolidation  of  the  past  20 
years  has  been  explained  as  a  result  of 
decreased  defense  spending  or  capacity 


underutilization  (e.g.,  Augustine,  1997; 
Deutch,  2001)  and  has  been  accompanied 
by  concerns  over  the  level  of  competition 
between  remaining  prime  contractors 
(Ricks,  1996).  Often  the  defense  industry 
is  compared  with  a  competitive  market  of 
common  commercial  products  and  as¬ 
sumed  to  be  inefficient.  However,  com¬ 
petitive  markets  of  common  commercial 
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products  display  multiple  competitors  and 
customers  where  prices  can  be  predicted 
from  given  levels  of  supply  and  demand* 
Meanwhile,  a  defense  market  with  differ¬ 
entiated  weapon  systems  as  products  dis¬ 
plays  important  differences  from  a  classic 
competitive  market  (Peck  &  Scherer,  1962; 
Sapolsky  &  Gholz,  1998),  and  it  should 
be  recognized  that  assumptions  about  the 
applicability  of  competitive  marketplaces 
can  result  in  dubious  policy  recommenda¬ 
tions  (King  &  Driessnack,  2003;  Langlois 
&  Robertson,  1995). 

Traditional  explanations  for  the  con¬ 
solidation  of  the  defense  industry  based 
on  decreased  defense  spending  are  not 
consistent  with  a  long-term  view  of  the 
industry.  Although  recent  defense  indus¬ 
try  consolidation  is  unprecedented,  the  U.S. 
defense  budget  has  followed  a  cyclical 


pattern  with  both  decreases  and  in¬ 
creases  in  spending  since  1 952  (see  Fig¬ 
ure  1).  This  suggests  that  the  most  recent 
defense  industry  consolidation  has  been 
driven  by  factors  beyond  decreased  de¬ 
fense  spending.  Markusen’s  (1998)  pre¬ 
diction  that  surviving  defense  firms  would 
become  smaller  and  focus  more  on  com¬ 
mercial  markets  has  not  come  to  pass.  For 
example,  while  Boeing  offers  both  com¬ 
mercial  and  defense  products,  it  grew 
larger  through  its  acquisition  of  McDonnell 
Douglas,  which  increased  its  reliance  on 
defense.  Additionally,  although  mergers 
have  consolidated  the  number  of  firms  and 
decreased  the  number  of  personnel  em¬ 
ployed  by  defense  firms,  Department  of 
Defense  (DoD)  Industrial  Policy  reports 
consistently  show  industry  capacity  has  not 
undergone  equivalent  reductions. 


Year 


Figure  1*  Defense  Spending  1952-1997  in  FY1996  Dollars 
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Peck  and  Scherer  (1962)  cautioned 
against  applying  traditional  assumptions 
of  a  competitive  price-driven  market  to  the 
defense  industry  as  their  applicability  is 
tenuous,  at  best.  The  defense  market  is 
unique  and  displays  an  increased  role  of 
government  where  it  acts  as  both  a  buyer 
of  goods  and  regulator  of  the  market. 
Government  actions  predominate  the 
defense  market  as  it  sets  the  rules  (e.g., 
regulates  contract  types)  and  alone  is 
responsible  for  uncertainty  of  demand  as 
the  sole  buyer  of  defense  goods.1  Gov¬ 
ernment  represents  an  active  institution  in 
the  defense  industry,  and  institutions  con¬ 
tribute  to  market  structure  by  defining 
transaction  costs  (Hoskisson,  2000).  The 
neo-classical  economic  view,  which  is 
often  the  basis  of  current  industrial  poli¬ 
cies,  minimizes  the  role  of  transaction 
costs  and  the  impact  of  institutions  on 
firms. 

Our  contribution  involves  applying  an 
evolving  framework  called  New  Institu¬ 
tional  Economics  (Greenwood  &  Hinings, 
1996;  North,  1999;  Williamson,  1985)  to 
the  defense  industry.  The  result  is  an  ini¬ 
tial  assessment  of  the  impact  of  institu¬ 
tions  in  changing  transaction  costs,  and 
the  potential  relationship  these  changes 
played  in  the  recently  observed  defense 
industry  consolidation.  Existing  research 
has  recognized  that  transaction  costs  play 
a  role  in  defense  markets.  For  example, 
Rogerson  (1994)  recognized  the  key  role 
of  government  in  shaping  the  defense 
procurement  process  and  unique  charac¬ 
teristics  that  impact  weapons  procurement 
incentives.  He  goes  on  to  lament  that 
economics  has  largely  been  absent  from 
shaping  defense  industry  institutions  and 
its  regulatory  environment  because  of  the 
unique  nature  of  defense  procurement. 


Our  goal  is  to  facilitate  a  better  under¬ 
standing  of  the  defense  industry  and  its 
efficiency  by  providing  a  framework  that 
allows  more  informed  policy  recom¬ 
mendations  through  a  better  explanation 
of  the  role  of  institutions  in  the  defense 
industry. 


Defense  Industry  Consolidation 


Our  analysis  begins  with  a  summary  of 
the  defense  industry.  We  define  defense 
films  similar  to  Chu  and  Waxman  (1998) 
as  films  that  have  established  capabilities 
and  competencies  in  dealing  with  the  DoD. 
An  important  implication  of  this  definition 
is  that  defense  firms  have  evolved  to  be¬ 
come  specialized  in  the  sense  that  they 
focus  on  the  transactions  with  a  monop¬ 
sony  customer — the 

agencies  and  organiza¬ 
tions  in  the  DoD  and  not 
in  any  particular  product 
or  technology.  Our  ini¬ 
tial  look  focuses  on  the 
top  of  the  defense  indus¬ 
try  hierarchy,  the  5 
firms,  Lockheed  Martin, 

Boeing,  Raytheon, 

Northrop  Grumman,  and  General  Dynam¬ 
ics,  which  have  consolidated  to  increase 
both  their  share  of  the  defense  business 
and  percentage  of  their  business  dependent 
on  defense  work. 

These  defense  firms  are  public  firms, 
but  many  of  the  similarities  to  commercial 
firms  operating  in  the  U.S.  market  end 
there.  Defense  firms  sell  unique  products 
in  a  monopsony  where  the  only  buyer  is 
the  U.S.  government.  Contract  competi¬ 
tions  typically  involve  situations  where 
‘winner  takes  all,’  and  R&D  costs  for 


"Defense  firms 
sell  unique 
products  in  a 
monopsony 
where  the  only 
buyer  is  the  U L$« 
government." 
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their  high-technology  products  are 
largely  subsidized  (McNaughter,  1989; 
Peck  &  Scherer,  1962).  Viewing  defense 
firms  as  operating  in  a  traditional  com¬ 
mercial  market  would 
require  that  the  special¬ 
ized  knowledge  for  do¬ 
ing  business  with  the 
government  could  be 
easily  obtained  (Chu  & 
Waxman,  1998).  Fur¬ 
ther  understanding  the 
defense  industry  con¬ 
solidation  requires  un¬ 
derstanding  a  market  defined  by  the  trans¬ 
actions  with  U.S.  government  procure¬ 
ment  offices  utilizing  highly  stylized  rules 
for  contracting.  We  employ  New  Institu¬ 
tional  Economics  (NIE)  to  examine  the 
role  technology  and  institutions  played 
in  defense  industry  consolidation. 


"There  ere 
multiple  unique 
technologies 
required  for 
different  catego¬ 
ries  of  weapon 
systems*" 


New  Institutional  Economics 


The  basic  unit  of  analysis  or  core  of  the 
NIE  framework  Is  transaction  costs  and 
how  they  are  determined  by  the  interac¬ 
tion  of  changing  institutions,  environment, 
and  enforcement  arrangements  of  formal 
and  informal  market  rules.  When  exam¬ 
ining  transaction  costs,  the  question  that 
needs  to  be  resolved  Is  on  what  principal 
dimension  does  a  transaction  differ 
(Williamson,  1996).  Additionally,  Wil¬ 
liamson  indicates  that  similar  effort  is 
needed  to  understand  differences  in 
governance  structures  that  bring  order  to 
transactions. 

Our  analysis  begins  by  examining 
how  defense  industry  transactions 
changed  as  a  result  of  the  advancement 
in  technology  and  the  need  for  integrated 


weapons,  evolving  government  institu¬ 
tions,  and  impact  of  winner  take  al  con¬ 
tracts.  Again,  our  goal  is  to  see  how  chang¬ 
ing  institutions  and  transaction  costs  can 
also  help  explain  defense  industry  con¬ 
solidation.  We  do  not  claim  that  these 
forces  are  the  only  ones  acting  on  defense 
firms.  Instead,  we  highlight  that  they  rep¬ 
resent  contributing  forces  that  have  not 
been  previously  examined,  in  compari¬ 
son  to  decreased  procurement  funding. 
Our  findings  suggest  that  changing  insti¬ 
tutions  and  transaction  costs  provides  a 
more  complete  story  of  the  forces  driv¬ 
ing  defense  industry  consolidation. 


Evolving  Technology 


Technological  change  can  have  exten¬ 
sive  impacts  on  the  competitive  dynamics 
of  industries  (Anderson  &  Tushman, 
1990).  There  are  multiple  unique  tech¬ 
nologies  required  for  different  categories 
of  weapon  systems.  We  focus  on  provid¬ 
ing  examples  from  fighter  aircraft  where 
significant  advances  in  core  and  support¬ 
ing  technologies  can  be  observed  to 
demonstrate  that  maturing  technology  has 
contributed  to  defense  industry  consoli¬ 
dation.  As  technology  matures,  a  domi¬ 
nant  design  is  established,  and  there  Is 
pressure  for  firms  to  consolidate,  as  fewer 
product  offerings  exist  in  a  market  (King, 
Covin  &  Hegarty,  2003). 

In  the  case  of  fighter  aircraft  technol¬ 
ogy,  following  World  War  n,  technology 
evolved  from  reciprocating  engines  to  jet 
engines  driving  changes  to  basic  aircraft 
design.  Uncertainty  about  potential  perfor¬ 
mance  and  product  design  drove  a  large 
number  of  unique  aircraft  designs  from 
multiple  defense  firms.  For  example,  a 
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total  of  nine  prime  contractors  designed 
and  flew  40  different  fighter  aircraft 
designs  during  the  1940s  and  1950s 
(Lorell  &  Levaux,  1998).  However,  as  jet 
fighter  technology  matured  the  number 
of  aircraft  designs  fell  with  less  than  a 
dozen  U.S.  fighter  aircraft  developed 
since  1960.  Maturing  technology  contrib¬ 
utes  to  pressures  on  major  defense  firms 
to  consolidate  as  fewer  aircraft  designs  re¬ 
quired  fewer  prime  contractors  that  are  ca¬ 
pable  of  integrating  and  manufacturing 
fighter  aircraft.  A  related  observation  that 
supports  this  statement  is  that  there  have 
been  no  new  entrants  into  U.S.  manned 
aircraft  production  since  1945  (Birkler, 
Bower,  Drezner,  Lee,  Lorell,  Smith, 
Trimble,  &Younossi,  2003).  Additionally, 
the  aircraft  types  that  are  produced  are 
maintained  longer  and  upgraded  more  of¬ 
ten.  Upgrade  contracts  are  typically 
awarded  to  the  original  prime  contractor, 
requiring  defense  firms  to  retain  a 
workforce  able  to  manage  multiple  tech¬ 
nology  insertion  programs. 

The  award  of  the  F-22  Raptor  and  F-35 
Joint  Strike  Fighter  contracts  to  Lockheed 
Martin  has  resulted  in  a  situation  where 
no  other  new  manned  fighter  aircraft 
program  is  in  design  or  planned  to  be  in 
design  for  at  least  the  next  10  years.  This 
reduction  in  the  number  of  aircraft  de¬ 
signs  may  be  far  more  influential  on  in¬ 
dustry  consolidation  than  decreased  de¬ 
fense  spending.  For  example,  the  Joint 
Strike  Fighter  is  the  largest  DoD  program 
to  date  in  terms  of  its  anticipated  budget 
value.  The  consolidation  of  aircraft  type 
is  also  consistent  with  bomber  and  cargo 
aircraft.  We  maintain  this  consolidation  of 
weapon  systems  influenced  industry  con¬ 
solidation  more  than  the  cycles  in  defense 
spending.  However,  institutional  factors 


also  contributed  to  defense  industry 
consolidation. 


In  looking  at  the  consolidation  in  the 
defense  industry  it  is  important  to  consider 
the  changing  institutions.  Institutions  are 
the  formal  and  informal  rules  along  with 
their  enforcement  arrangements  that  influ¬ 
ence  the  nature  of  the  transactions  and  their 
costs  (Furubotn  &  Richter,  1998).  Com¬ 
bined  with  evolving  technology,  chang¬ 
ing  institutions  have  influenced  market 
forces  within  the  defense  industry.  We  sum¬ 
marize  the  forces  that  have  influenced  de¬ 
fense  firms  with  the  goal  of  providing  in¬ 
sight  on  the  drivers  of  defense  industry  con¬ 
solidation  beyond  < 
spending.  Specifi¬ 
cally,  the  evolution 
of  government  pro¬ 
curement  practices 
has  likely  impacted 
consolidation 
within  the  defense 
industry.  We  exam¬ 
ine  the  govern¬ 
ment  procurement 
practices  in  three 
areas — defense  pro¬ 
curement  policy,  re¬ 
quirements  gen¬ 
eration,  and  the 
organization  of  government  procurement 
agencies — that  shape  the  interaction  of  ex¬ 
changes  between  the  government  and  de¬ 
fense  contractors. 

Procurement  Policy 

Since  WWII,  a  set  of  diverse  organiza¬ 
tions  in  each  service  evolved  from  their 


Changing  Institutions 


:reased  defense 


'Maturing  technol- 
>gy  contributes  to 
pressures  on  major 
lefense  firms  to 
sonsolidate  as  fewer 
aircraft  designs 
required  fewer 
prime  contractors 
that  are  capable  of 
integrating  and 
manufacturing 
Fighter  aircraft." 
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various  histories  to  manage  weapons 
system  procurement.  The  government 
did  not  buy  as  a  monopsony,  instead  nu¬ 
merous  organizations  for  each  service 
acted  as  independent  actors  on  the  mar¬ 
ket  for  particular  service  weapons.  The 
policies  of  these  organizations  have 
evolved  to  become  unified  with  the  de¬ 
velopment  of  common  policies  and  regu¬ 
lations.  Government  procurement  policies 

"In  general, 

20  years  ago 
contractors  spe¬ 
cialized  not  only 
on  a  particular 
weapon  system, 
but  also  on  the 
unique  procure¬ 
ment  organization 
within  each  armed 
service," 


dustry. 

The  FAR  has  evolved  as  an  all-encom¬ 
passing  regulation  governing  federal 
procurement  in  the  past  20  years — a 
period  that  aligns  with  the  defense 
industry's  consolidation.  The  origin  of  the 
present  day  FAR  was  the  Armed  Services 
Procurement  Act  of  1947  (Nagle,  1999). 
Following  that  act  the  number  and  size  of 
regulations  governing  procurement 
steadily  grew.  By  1979,  there  were  877 
different  sets  of  procurement  regulations 
including  directives,  bulletins,  and  in¬ 
structions,  comprising  64,600  pages  of 
regulations.  Since  then,  the  specific  poli¬ 
cies  for  each  armed  service  and  the  vari¬ 
ous  other  non-DoD  government  agencies 
have  been  consolidating  into  a  single  fed¬ 
eral  regulation.  Progress  has  been  made 


and  regulations  are  out¬ 
lined  in  the  DoD  5000 
series  regulations  and 
contractual  manage¬ 
ment  procedures  (the 
interface  with  commer¬ 
cial  industry)  in  the 
form  of  the  Federal  Ac- 
quisition  Regulation 
(FAR).  These  procure¬ 
ment  practices  define 
the  interaction  between 
the  government  and 
firms  in  the  defense  in¬ 


to  establish  the  FAR  as  the  single  federal 
procurement  policy  (Nagle,  1999). 

In  general,  20  years  ago  contractors 
specialized  not  only  on  a  particular  wea¬ 
pon  system,  but  also  on  the  unique  pro¬ 
curement  organization  within  each  armed 
service.  For  example,  as  recently  as  the 
1980s,  each  Service  had  separate  con¬ 
tract  monitoring  processes  that  drove 
firms  to  specialize  in  dealing  with  indi¬ 
vidual  armed  services.  If  a  firm  did 
business  with  multiple  branches  of  the 
military,  it  was  not  unusual  for  each  service 
to  have  its  own  contract  monitoring 
personnel  assigned  to  a  firm's  plant  full 
time. 

This  has  changed  over  the  past  20  years 
with  a  series  of  unifying  events.  One 
example  is  the  single-process  initiative 
(SPI)  that  began,  in  1994,  and  in  a  matter 
of  two-years  time  relieved  defense  firms 
from  multiple  processes  driven  by  numer¬ 
ous  government  procurement  offices.  The 
oversight  was  placed  under  a  single 
defense  agency,  Defense  Contract  Man¬ 
agement  Agency  (DCMA),  eliminating 
almost  all  of  the  unique  service-oriented 
contract  monitoring  offices. 

The  creation  of  a  single  office  fur¬ 
ther  contributed  to  the  standardization 
of  government  processes.  It  should  be 
noted  that  the  single  process  initiative 
did  not  eliminate  the  unique  require¬ 
ments  of  the  government,  but  reduced 
the  ability  of  the  different  DoD  agen¬ 
cies  and  services  to  require  unique  pro¬ 
cesses.  The  contractor  could  have  a 
single  set  of  processes  for  a  given  fa¬ 
cility  or  across  multiple  facilities.  As 
government  procurement  became 
more  standardized,  defense  firms  were 
less  constrained  to  specializing  for 
each  armed  service — widening  their 
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opportunities  to  take  advantage  of  their 
specialization.  This  may  have  contributed 
to  defense  industry  consolidation  in  that 
niche  firms  may  have  decided  to  exit  the 
industry  as  they  realized  their  niche  would 
be  encroached  upon  as  uniform  govern¬ 
ment  procurement  practices  increased 
competition  and  reduced  transaction  costs 
required  to  deal  with  multiple  armed  ser¬ 
vice  weapons  procurement  organizations. 

The  same  consolidation  and  standard¬ 
ization  that  happened  on  the  FAR  also  oc¬ 
curred  with  the  DoD  5000  series;  the  policy 
focused  on  government  procurement  pro¬ 
cesses  and  thus  influenced  the  structure  of 
the  procurement  organizations.  The  5000 
series  was  first  issued  in  July  1971,  as  an 
initiative  of  then  Deputy  Secretary  of  De¬ 
fense  David  Packard  (Przemieniecki, 
1993)  with  the  primary  theme  of  central¬ 
ized  policy  and  decentralized  execution 
(Ferrara,  1996).  The  initial  focus  was  on 
the  procurement  of  major  weapon  systems, 
allowing  the  services  to  continue  to  acquire 
non-major  systems  under  their  own  poli¬ 
cies.  However,  in  1987,  the  5000  series 
was  extended  to  all  procurement  programs, 
and  consolidated  over  60  different  direc¬ 
tives,  instructions  and  memoranda.  The 
5000  series  has  continued  to  reduce  the 
number  of  unique  armed  services  pro¬ 
cesses.  This  reduction  along  with  the 
evolving  technology  and  move  toward  joint 
programs  (more  than  one  armed  service 
involved)  has  moved  the  various  armed 
services  toward  a  single  monopsony 
(single  customer)  versus  a  collection  of 
monopsony  buying  related  but  not  the 
same  products.  An  example  is  the  past  pur¬ 
chase  of  fighter  aircraft  by  the  Navy,  like 
F/A-18s,  and  the  Air  Force,  like  F-16s,  to 
the  joint  purchase  of  the  F-35  Joint  Strike 
Fighter. 


Requirements  Generation 

The  requirements  generation  process 
establishes  the  collective  intent  of  the 
government  on  what  weapon  systems  are 
purchased.  Requirements  generation 
operates  in  conjunction  with  the  Planning, 
Programming,  Budgeting,  and  Execution 
(PPBE)  process  used  to  manage  resources 
by  applying  constraints.  While  internal  to 
the  government,  requirements  generation 
is  visible  to  industry  and  provides  infor¬ 
mation  on  the  future  intent  of  the  govern¬ 
ment-buying  preferences.  Government 
selection  and  procurement  of  given 
weapon  systems  is  not  independent  of 
evolving  technology. 

The  number  of  aircraft  ultimately  pur¬ 
chased  has  fallen  as  technological  ad¬ 
vances  deliver  additional  capability.  For 
example,  by  1958,  when  the  F-4  fighter 
prototype  first  flew  (Smith  &  Friedman, 
1980),  the  speed  of  fighter  aircraft 
reached  a  plateau  and 
technology  changed 
to  focus  on  guided 
missiles,  which  repre¬ 
sented  a  growing  per¬ 
centage  of  aerospace 
industry  sales  and  in 
many  cases  were 
viewed  as  a  substitute 
for  aircraft  (Simonson, 

1968).  The  increased  capability  of  air¬ 
craft  from  technology  advances,  such  as 
guided  weapons,  simply  translated  into  a 
requirement  for  fewer  numbers  of  aircraft. 
For  example,  the  impact  is  clearly  dem¬ 
onstrated  in  comparing  U.S.  Air  Force 
procurement  quantities  of  F-4  aircraft 
(2,600)  to  F-117  stealth  fighters  (59). 
However,  the  resulting  capability  is  com¬ 
parable  in  that  a  single  F-117’s  bombing 
effectiveness  equates  to  95  F-4  aircraft 


"The  number  of 
aircraft  ultimately 
purchased  has 
fallen  as  techno¬ 
logical  advances 
deliver  additional 
capability" 


69 


Defense  Acquisition  Review  Journal — January-April  2004 


(Toffler  &  Toffler,  1993).  The  general 
impact  of  available  technology  deliv¬ 
ering  more  capable  weapon  platforms 
suggests  fewer  major  defense  firms  were 
needed  (Deutch,  2001).  Additionally,  an 
increased  emphasis  on  jointness  has  led 
to  the  Joint  Strike  Fighter  becoming  the 
single  planned  replacement  aircraft  for 
the  Air  Force’s  F-16,  the  Navy’s  F/A- 
18,  and  the  Marine’s  AV-8B. 

Further  decreasing  the  number  of  dif¬ 
ferent  aircraft  designs,  the  trend  in  require¬ 
ments  generation  has 
been  toward  joint  pro¬ 
grams,  or  programs  that 
meet  the  needs  of  more 
than  one  armed  service. 
Emphasis  on  joint 
weapon  programs  has 
steadily  increased  since 
the  F-lll  aircraft  was 
originally  designed  to 
meet  the  needs  of  two 
services — the  Air  Force 
and  Navy  (Smith  &  Friedman,  1980). 
Among  other  initiatives,  the  Goldwater- 
Nichols  Department  of  Defense  Reorga¬ 
nization  Act  of  1986  made  the  Joint  Chiefs 
of  Staff  advocates  for  a  joint  military  per¬ 
spective  with  the  vice  Chairman  of  the 
Joint  Chiefs  of  Staff  ( VCJCS)  responsible 
for  chairing  the  Joint  Requirements 
Oversight  Council  (JROC),  a  special 
council  on  military  requirements  (Owens, 
1994).  After  the  termination  of  the  Navy’s 
A- 12  aircraft  program,  in  1991,  joint  pro¬ 
grams  took  hold  as  the  dominant  paradigm 
as  the  termination  essentially  ended 
service-  and  mission-unique  aircraft 
programs  (Jefferson,  1991).  The  empha¬ 
sis  on  joint  programs  has  also  increased 
the  monopsony  power  of  the  government 


as  procurement  is  further  centralized 
from  the  different  armed  services. 

Another  consideration  is  that  each 
aircraft  program  essentially  involves  a 
“ winner  take  all”  competition  for  manu¬ 
facturers.  The  most  recent  fighter  aircraft 
competition  involved  the  Joint  Strike 
Fighter  (JSF),  and  the  associated  devel¬ 
opment  contract  was  awarded  to  Lock- 
heed-Martin  and  Boeing  in  November 
1996  ( Wall  Street  Journal ,  1996a).  One 
month  later,  after  just  ending  similar 
merger  discussions  only  six  months  ear¬ 
lier,  Boeing  announced  a  merger  with 
McDonnell  Douglas,  the  loser  of  the  JSF 
competition  ( Wall  Street  Journal,  1996b). 
The  only  change  in  prospects  for 
McDonnell  Douglas,  after  it  scuttled 
merger  discussions  with  Boeing  earlier  the 
same  year,  was  the  JSF  contract  award. 
As  the  JSF  represented  the  only  major 
fighter  aircraft  contract  anticipated  for  at 
least  a  decade,  consolidation  represented 
a  reasonable  reaction.  The  JSF  program 
also  demonstrates  the  level  of  emphasis 
on  joint  programs.  The  JSF  is  the  planned 
replacement  for  the  Navy  F/A-18,  Air 
Force  F-16  and  Marine  AV-8B  aircraft, 
plus  aircraft  for  several  foreign  military 
partners  that  are  participating  in  the 
program.  Maturing  technology,  winner 
take  all  competitions,  and  increased 
emphasis  on  joint  programs  all  contrib¬ 
uted  to  fewer  aircraft  programs  that,  in 
turn,  drove  consolidation  as  each  factor 
signaled  a  requirement  for  fewer  aircraft 
manufacturers. 

Government  Organization 

The  most  existing  literature  on  the  de¬ 
fense  industry  overlooks  the  organization 
of  government  agencies.  However,  due  to 


"Another  consider¬ 
ation  is  that  each 
aircraft  program 
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volves  a  *  winner 
take  eiP  competi¬ 
tion  for  manufac¬ 
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the  interaction  between  government  and 
defense  contractor  offices,  changes  to  the 
structure  of  government  procurement 
organizations  also  impacts  defense  firms. 
Procurement  organizations  have  under¬ 
gone  two  primary  changes — centralization 
and  downsizing. 

The  centralization  of  procurement 
policy  and  intervention  by  Congress 
through  the  Goldwater-Nichols  Depart¬ 
ment  of  Defense  Reorganization  Act  of 
1986,  created  consistent  government  pro¬ 
curement  oversight  by  establishing  Pro¬ 
gram  Executive  Officers  (PEOs).  The  PEO 
structure  for  the  majority  of  procurement 
programs  established  a  streamlined  autho¬ 
rity  through  the  civilian  leadership  from 
the  Service  Acquisition  Executive  (SAE), 
the  most  senior  service  official  for  acqui¬ 
sitions  of  weapon  systems,  to  each  weapon 
system  program  manager.  This  change 
lessened  the  influence  of  other  varied 
organizations  in  the  DoD  and  respective 
services  from  the  day-to-day  operations 
of  weapons  system  procurement. 

Reduced  defense  budgets  have  placed 
an  emphasis  on  downsizing  of  government 
procurement  agencies  that  are  largely 
viewed  as  ancillary  to  the  armed  services 
mission  of  winning  wars.  Traditionally 
government  program  offices  issued  vari¬ 
ous  contracts  to  different  prime  contrac¬ 
tors  for  different  subsystems  of  an  overall 
weapon  system  and  the  government  acted 
as  the  integrator.  However,  reduced  per¬ 
sonnel  resources  following  the  end  of  the 
Cold  War  led  to  government  program  of¬ 
fices  placing  more  effort  on  contract,  re¬ 
sulting  in  a  migration  of  tasks  and  associ¬ 
ated  transactions  from  the  government  to 
defense  firms  acting  as  prime  contractors. 

This  migration  of  responsibility  was 
generally  known  as  Total  System  Program 


Responsibility  (TSPR).  Under  this  con¬ 
cept  a  defense  firm  selected  as  the  prime 
contractor  for  a  program  was  given  this 
“total  responsibility”  under  broad  integra¬ 
tion  contracts.  The  prime  contractor  is  now 
often  responsible  for  integration  of  an 
overall  system,  instead  of  the  govern¬ 
ment’s  weapon  system  program  office. 
Increased  responsibility  by  prime  contrac¬ 
tors  had  the  effect  of  limiting  the  number 
of  contracts  available  from  the  government 
as  subsystem  contracts  previously  issued 
by  the  government 
were  bundled  within 
a  prime  contractor’s 
weapon  system  con¬ 
tract.  The  combined 
effects  of  centraliza¬ 
tion  and  downsizing 
led  to  a  shift  in 
workload  from  govern¬ 
ment-buying  agents  to 
prime  defense  con¬ 
tractors,  and  reduced 
government  unique 
transaction  costs  with 
the  market.  The  com¬ 
petitive  impact  of  this 
shift  in  subcontract  management  from 
the  government  to  defense  firms  is  un¬ 
known,  and  represents  an  opportunity 
for  future  research. 

While  the  number  of  government 
procurement  personnel  has  decreased, 
there  was  increased  emphasis  on  ensuring 
they  were  better  trained.  Congress  passed 
the  Defense  Acquisition  Workforce  Im¬ 
provement  Act  (DAWIA)  in  1990,  and  it 
outlined  education  and  certification 
procedures  for  a  professional  procurement 
workforce.  Centralization  of  procurement 
policies  was  followed  by  centralization  of 
training,  and  in  1992  Defense  Acquisition 


"Reduced 
defense  budgets 
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University  (DAU)  was  formed.  The 
consolidation  of  the  training  for  govern¬ 
ment  personnel  on  the  unique  institutional 
mechanisms  and  structures  further  dem¬ 
onstrated  government  procurement  policy 
centralization  that  was  helping  to  prompt 
defense  industry  consolidation.  The  con¬ 
solidation  of  processes  would  only  he 
strengthened  as  government  procurement 
personnel  received  training  on  uniform 
procurement  practices  from  a  centralized 
training  organization. 


Summary 


Forces  relating  to  changing  technology 
and  changing  government  institutional 
practice  corresponded  with  decreased  de¬ 
fense  spending  to  drive 
defense  industry  consoli¬ 
dation.  Changing  gov¬ 
ernment  practice  includes 
joint  procurement,  cen¬ 
tralization  of  procurement 
policy,  standardization  of 
government  procurement 
organization,  and  stan¬ 
dardized  training  of  a 
core  of  professional  pro¬ 
curement  personnel.  To¬ 
gether  these  evolving 
changes  led  to  common 
government  policies  that 
changed  the  dynamic  in  the  defense  in¬ 
dustry  and  pressured  defense  firms  to  re¬ 
duce  transaction  costs  through  consolida¬ 
tion.  Viewing  the  defense  industry  consoli¬ 
dation  using  this  framework  provides  a 
more  complete  explanation  than  reduced 
defense  spending.  The  framework  also 
provides  a  better  foundation  for  future 
policy  recommendations. 


"Together  these 
evolving  changes 
led  to  common 
government 
policies  that 
changed  the 
dynamic  in  the 
defense  industry 
and  pressured 
defense  firms  to 
reduce  transac¬ 
tion  costs  through 
consolidation/' 


Discussion 

The  central  message  of  this  paper  is  that 
the  U.S.  government  holds  a  unique  po¬ 
sition  in  the  defense  industry  as  both  a 
monopsony  customer  and  as  a  federal 
government  with  regulatory  oversight  that 
controls  the  mechanisms  in  the  market. 
This  position  allows  the  government  to 
have  a  hyper  influence  on  the  institutions 
governing  the  mechanisms  of  exchange 
and  thus  the  structure  of  the  defense 
industry.  Evolving  government  policies 
and  their  impact  on  transaction  costs 
(North,  1990)  brings  clarity  to  explain¬ 
ing  recent  defense  industry  consolidation. 

Our  observations  are  consistent  with 
views  of  institutions  impacting  transaction 
costs  and  the  structure  of  markets.  We  find 
that  both  government  procurement  orga¬ 
nizations  and  defense  firms  should  be 
viewed  as  rationally  reducing  transaction 
costs  and  thus  the  structure  of  the  defense 
market.  The  defense  industry  experiences 
a  unique  set  of  transaction  costs  from  those 
experienced  in  commercial-oriented  free 
markets,  and  the  application  of  competi¬ 
tive  market  prescriptions  focused  on  prices 
to  the  defense  industry  is  inappropriate. 

An  institutional  framework  that  considers 
transaction  costs  provides  a  more  complete 
picture  for  moving  forward  and  assessing 
defense  industry  efficiency  issues  that  con¬ 
cern  policy  makers.  The  current  view  com¬ 
monly  held  within  the  government  and  the 
literature  focuses  on  competition  driving 
defense  industry  efficiencies  and  prevent¬ 
ing  defense  firms  from  collecting  mo¬ 
nopoly  rents.  This  view  is  derived  from 
traditional  supply  and  demand  models  with 
multiple  suppliers  and  customers  and  has 
less  application  considering  the  unique 
nature  of  the  defense  industry.  For  ex¬ 
ample,  the  Truth  in  Negotiation  Act 
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(TINA)  of  1962  and  creation  of  the  De¬ 
fense  Contract  Audit  Agency  in  1965 
(Lorell,  et  al. ,  2000)  has  created  a  system 
where  the  government  provides  a  coun¬ 
terbalance  to  any  monopoly  power,  en¬ 
suring  it  only  pays  fair  and  reasonable 
costs  for  products. 

The  changes  in  the  defense  industry 
over  the  past  20  years  represent  the  gov¬ 
ernment  and  firms  rationally  reducing 
transaction  costs  through  centralization 
and  consolidation  respectively,  but  not  a 
substantial  increase  use  of  free  market 
mechanisms.  It  is  unlikely  that  the  defense 
industry  will  ever  approximate  a  competi¬ 
tive  market,  as  long  as  the  government 
remains  a  monopsony  customer  with 
regulatory  oversight. 

Any  application  of  a  competitive 
market  framework  will  shed  limited  light 
on  the  realities  of  defense  industry  struc¬ 
ture  and  its  evolution.  We  show  that  ex¬ 
panding  the  view  of  the  defense  industry 
to  include  institutions  and  related  trans¬ 
action  cost  provides  improved  explana¬ 
tion  for  observed  defense  industry  phe¬ 
nomenon,  or  in  the  present  case  industry 
consolidation.  Institutions  and  transaction 
costs  can  provide  a  more  realistic  frame¬ 
work  for  economic  analysis,  and  should 
play  a  more  active  role  in  framing  policy 
recommendations. 

The  defense  industry  is  typically  char¬ 
acterized  as  inefficient.  However,  deter¬ 
mining  the  efficiency  of  a  market  needs 
to  consider  constraints  imposed  by  trans¬ 
action  costs,  and  an  outcome  without  a 
feasible  and  superior  alternative  should 
be  accepted  as  efficient  (Williamson, 
1985).  Th q  feasibility  of  alternatives  needs 
to  be  understood  within  the  institutions 
relating  to  a  market,  or  relevant  legal,  eco¬ 
nomic,  and  political  realities,  which  in  the 


case  of  the  defense  industry,  play  a  larger 
role  than  a  traditional  competitive  mar¬ 
ketplace.  For  activities  in  the  public  sec¬ 
tor,  external  costs  imposed  on  the  operation 
of  markets  may  be  higher  than  necessary — 
reducing  these  costs  requires  modifying 
the  institutions  governing  decision¬ 
making  heuristics  (Buchanan  &  Tullock, 
1962).  Indeed,  the  preoccupation  with 
rational  choice  and  efficient  market  often 
blind  people  to  the  implications  of 
complex  environments  and  the  realities  of 
incomplete  information  (North,  1999). 
Recognizing  path  dependence  is  key  to 
understanding  long-run  economic  change, 
and  our  review  shows  the  additional  forces 
that  evolving  government  institutions 
placed  on  defense  firms  likely  contributed 
to  consolidation. 

In  closing,  reduced  defense  spending 
alone  does  not  fully  explain  the  consoli¬ 
dation  of  defense  firms  witnessed  during 
the  1990s,  as  decreased  defense  spending 
has  occurred  in  the  past  without  similar 
consolidation.  The  unique  nature  of  the 
defense  industry  makes  the  application  of 
traditional  price  driven  explanations  and 
associated  policy  recommendations  from 
competitive  markets  tenuous.  Examination 
of  the  interaction  of  evolving  technology, 
changing  institutions,  to  include  the  pro¬ 
curement  policy,  weapons  requirements 
process,  and  the  procurement  organizations 
on  transaction  costs  in  the  defense  indus¬ 
try  will  more  fully  explain  the  forces  driv¬ 
ing  consolidation  and  provide  greater  in¬ 
sight  for  policy  makers  seeking  to  improve 
the  performance  of  the  defense  industry. 
Further  research  is  needed  to  develop  a 
robust  institutional  framework  of  the  de¬ 
fense  industry  and  the  related  government 
agencies. 
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IndNote 


1 .  Even  Foreign  Military  Sales  (FMS) 
typically  are  handled  through  the 
DoD,  or  U.S.  government. 
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TUTORIAL 


SURVEYING  COST  GROWTH 


Maj  Michael  Greiner,  USAF,  Capt  Vince  Sipple,  USAF, 
and  Mai  Edward  "Tony"  White,  USAF 

Cost  growth  that  weapon  systems  incur  throughout  their  acquisition  life  cycle 
concerns  those  who  work  in  the  acquisition  environment.  One  way  to  reduce 
the  amount  of  unexpected  cost  growth  is  to  develop  better  cost  estimates.  In 
attaining  better  cost  estimates  though,  it  is  often  helpful  to  understand  and 
account  for  potential  cost  drivers.  Several  cost  studies,  some  of  which  specifically 
focus  on  the  aircraft  industry,  have  been  performed  documenting  and 
investigating  these  growth  factors.  Overviews  of  these  various  cost  growth 
studies  are  presented  as  other  tools  for  the  cost  estimators  and  program 
managers. 


The  cost  growth  that  major  weapon 
systems  incur  represents  a  major 
management  challenge.  A  1993 
study  by  RAND  cites  that  by  the  time  a 
Department  of  Defense  (DoD)  Acquisi¬ 
tion  Category  (ACAT)  I  program  com¬ 
pletes  the  production  and  fielding  phase 
of  acquisition,  it  will  experience  an 
average  cost  growth  of  approximately  20 
percent  from  its  initial  estimate  (Drezner, 
Jarvaise,  Hess,  Hough,  &  Norton,  1993). 

During  the  early  eighties,  the  Reagan 
administration  recognized  two  ways  to 
control  the  problem  of  cost  growth  — 
perform  cost/requirements  tradeoffs  when 
costs  grow  and  create  better  estimates 
(Office  of  the  Under  Secretary  of  Defense, 
1981).  Since  then,  high-level  DoD 
management  personnel  continue  to  seek 
better  ways  of  controlling  cost  growth. 

While  program  managers  shoulder  the 
burden  of  controlling  cost  growth,  the 


second  method  promoted  by  the  Reagan 
administration  for  addressing  cost  growth, 
creating  more  realistic  estimates,  pertains 
more  directly  to  the  cost  estimating  com¬ 
munity  (Office  of  the  Under  Secretary  of 
Defense,  1981).  The  impediments  to 
creating  more  realistic  estimates  prima¬ 
rily  stem  from  the  many  uncertainties  that 
estimators  encounter  during  their  data 
collection  efforts.  The  Office  of  the 
Secretary  of  Defense  (OSD)  Cost  Analysis 
Improvement  Group  (CAIG)  gives  guide¬ 
lines  for  documenting  cost  estimating 
uncertainty  for  DoD  system  acquisition 
programs. 

First,  they  mandate  that  “areas  of  cost 
estimating  uncertainty  will  be  identified 
and  quantified”  (Department  of  Defense 
[DoD],  1992,  p.  22).  Programs  must 
document  this  uncertainty  in  the  Cost 
Analysis  Requirements  Document 
(CARD).  Second,  the  CAIG  prescribes 
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''the  use  of  probability  distributions  or 
ranges  of  cost”  to  quantify  uncertainty 
(DoD,  1992,  p.  22),  Third,  they  ask  that 
the  uncertainty  estimated  be  “attributable 
to  estimating  errors”  (DoD,  1992,  p.  22). 
For  instance,  they  list  as  such  examples, 
performance  and  weight  characteristics, 
new  technology,  manufacturing  initia¬ 
tives,  inventory  objectives,  schedules,  and 
financial  condition  of  the  contractor. 

DoD  procedures  also  provide  for  con¬ 
tingency  estimation  and  sensitivity  analy¬ 
sis,  giving  the  estimator  the  option  to  either 


include  or  exclude  a  contingency  amount. 
If  the  estimator  includes  contingency 
amount,  he  must  give  the  reason  for  the 
contingency  estimate  as  well  its  rationale. 
In  addition,  he  must  “include  an 
assessment  of  the  likelihood  that  the  cir¬ 
cumstances  requiring  the  contingency  will 
occur”  (DoD,  1992,  p.  22). 

To  better  address  uncertainty  in  the 
estimating  process,  the  defense  department 
sponsors  much  research.  For  this  article’s 
preparation,  we  reviewed  past  growth  stud¬ 
ies  in  the  literature  and  highlight  those  here 


Table  1*  Consolidated  Results  o#  Reviewed  Studies 


Study 

Main  Findings 

BMDO 

•Average  RDT&E  cost  growth  is  21%. 

•Average  Production  cost  growth  is  19%. 

•  Only  7  to  1 6%  of  programs  complete  at  or  below  target  cost. 

*  Programs  with  lower  dollar  value  have  greater  likelihood  of  cost  growth. 

RAND 

•  On  average  smaller  programs  have  more  cost  growth. 

•  RDT&E  funds  tend  to  experience  more  cost  growth  than  production  funds. 

•  Programs  maturity  affects  cost  growth.  Longer  implies  greater  likelihood. 

•  New-start  programs  over  modification  programs  have  more  cost  growth. 

NAVAIR 

*  Cost  growth  varies  across  different  cohorts  (grouped  by  different  estimate 
categories). 

*  No  statistical  significance  in  overall  cost  growth  due  to  program  size. 

*  Acquisition  changes  since  the  end  of  the  Cold  War  have  lead  to  less  cost  growth. 

*  Cost  growth  may  vary  by  commodity. 

CHRISTENSEN 
&  TEMPLIN 

*  Management  reserve  budgets  sensitive  to  contact  category  (fixed-price  higher 
than  cost  reimbursable). 

•  MR  budgets  do  not  differ  between  production  and  RDT&E  contracts. 

ESKEW 

•  Weight,  speed,  production  rate,  and  time  explain  more  than  90  percent  of  the 
variation  in  cost  growth  of  fighter  aircraft. 

D 

•  Urgency  of  the  program,  difficulty  of  technology,  and  degree  of  testing  affect  cost 
growth. 

•A  relationship  between  cost  growth  and  schedule  growth  in  both  the  development 
and  the  production  phases  exist. 

RAND-JSF 

*  Differences  between  the  competitive  and  non-competitive  development  and 
procurement  cost  growth  factors  are  not  statistically  significant. 
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that  we  feel  provides  the  cost  estimating 
community  with  an  overall  synopsis  of 
available  research.  We  list  and  explain 
those  in  detail  in  the  remaining  sections. 
For  ease  of  review  we  have  summarized 
their  major  findings  in  Table  1. 


Studies  of  Cost  Growth 
in  DoD  Acquisitions 


Ballistic  Missile  Defense 
Organization  Study 

A  recent  Ballistic  Missile  Defense 
Organization  (BMDO)  cost  growth  study 
provides  insight  into  the  nature  of  cost 
growth  in  DoD  programs.  Using  an  inter¬ 
nal  BMDO  database  of  programs  (created 
from  a  subset  of  the  Selected  Acquisition 
Report  [SAR]  database),  BMDO  finds  that 
Research  and  Development,  Test,  and 
Evaluation  (RDT&E)  cost  growth  averages 
21  percent  while  that  of  production  aver¬ 
ages  19  percent  (Coleman,  Summerville, 
DuBois,  &  Myers,  2000).  The  study  also 
shows  that  from  7  to  16  percent  of  pro¬ 
grams  complete  at  or  below  the  target  cost. 
Additionally,  data  from  the  study  suggests 
that  the  lower  the  dollar  value  of  a  pro¬ 
gram,  the  greater  likelihood  of  a  large  cost 
growth  factor.  Despite  this  trend,  though, 
the  authors  do  not  provide  any  statistical 
tests  to  explore  this  possibility. 

The  BMDO  researchers  further  note  that 
as  a  program  progresses,  cost  estimators 
revise  their  estimates,  thereby  reducing  the 
amount  of  estimated  risk  and  increasing 
the  amount  of  realized  risk.  Under  the  as¬ 
sumption  of  unbiased  risk  estimates,  one 
expects  realized  risk  to  equal  estimated  risk 
on  average,  given  a  large  sample.  How¬ 
ever,  the  study  shows  that  the  risk  portion 
of  the  estimate  decreases  more  slowly  than 


the  rest  of  the  estimate  increases.  This  evi¬ 
dences  a  general  trend  of  underestimating 
risk  (Coleman  et  al.,  2000). 

RAND  Study  (1993) 

The  study  canvassing  an  extensive  an*ay 
of  information  is  arguably  that  of  over  100 
data  points  performed  by  RAND.  Like  the 
BMDO  study,  RAND  uses  data  from  the 
SAR  reports,  and  RAND  focuses  on  the 
seven  categories  of  cost  variance  that  the 
SAR  contains:  quantity,  economic,  sched¬ 
ule,  engineering,  estimating,  support,  and 
other.  In  this  study,  RAND  finds  that  eco¬ 
nomic  and  quantity  changes  have  the  great¬ 
est  impact  on  cost  growth.  However,  RAND 
excludes  them  from  their  study  because 
these  two  factors  are  part  of  the  assump¬ 
tions  of  a  cost  estimate  initially. 

The  RAND  study  goes  on  to  relay  sev¬ 
eral  other  factors  that  relate  to  cost  growth. 
Like  the  BMDO  study,  RAND  finds  an  ap¬ 
parent  difference  in  cost  growth  based  on 
program  size.  That  is, 
smaller  programs  have 
on  average  more  cost 
growth  than  larger 
ones.  The  RAND  re¬ 
searchers  postulate  as 
the  reason  behind  this 
difference  the  greater 
level  of  management 
scrutiny  that  higher  dol¬ 
lar  programs  receive. 

This  stands  to  reason  considering  more 
management  scrutiny  should  translate  into 
better  cost  management. 

The  authors  of  the  RAND  study  offer 
another  possible  explanation  for  the  dif¬ 
ference  in  cost  growth  of  the  smaller  pro¬ 
grams,  “R&D  costs  are  a  large  portion  of 
total  costs  and  tend  to  incur  more  cost 
growth”  (Drezner  et  al.,  1993,  p.  49). 


"data  from  the 
study  suggests 
that  the  lower  the 
dollar  value  of  a 
program,  the 
greater  likelihood 
of  a  large  cost 
growth  factor." 
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In  other  words,  smaller  programs  are 
disproportionately  smaller  in  procure¬ 
ment  than  in  RDT&E.  Since  RDT&E 
funds  tend  to  experience  more  cost 
growth  than  procurement  funds,  the 
same  percent  RDT&E  cost  growth  in  a 
smaller  program  pushes  the  overall  pro¬ 
gram  percentage  cost  growth  higher 
than  a  large  program  counterpart. 

Program  maturity  also  factors  largely 
in  program  cost  growth.  The  RAND  study 
notes  that  “on  average,  cost  growth  in¬ 
creases  by  2.2  percent 
per  year  above  infla¬ 
tion  because  of  the  ef¬ 
fects  of  maturity” 
(Drezner  et  al.,  1993, 
p.  49).  RAND  empha¬ 
sizes  the  importance  of 
these  two  factors 
above  other  factors  in 
the  statement,  “pro¬ 
gram  size  and  maturity 
can  dominate  other 
factors  affecting  cost 
growth  outcomes  and  so 
must  be  considered  in 
both  the  analysis  and  the 
interpretation  of  re¬ 
sults”  (Drezner  et  ah,  1993,  p.  49). 

The  RAND  study  elucidates  the  impact 
of  new-start  programs  versus  modification 
programs,  finding  that  on  average,  the 
new-start  programs  experience  more  cost 
growth  than  modification  programs.  The 
RAND  study  also  finds  longer  programs 
to  have  more  cost  growth  than  shorter 
ones.  This  relationship  proves  intuitive: 
each  year  brings  the  opportunity  for  more 
cost  growth.  Interestingly,  they  find  no 
relationship  between  planned  length  and 
cost  growth  nor  between  schedule  slip 
and  cost  growth  (Drezner  et  al.,  1993). 


"While  the 
prototyping 
probably  does 
significantly 
reduce  risk,  it 
apparently  does 
not  reduce  if  to 
the  extent  that 
would  make  a 
prototyped 
program  have 
less  cost  growth 
than  a  non- 
prototyped 
program/* 


Finally,  RAND  finds  a  correlation 
between  prototyping  and  cost  growth: 

We  compared  the  cost  outcomes 
of  prototyping  and  nonproto¬ 
typing  programs,  expecting  to 
find  that  a  prototype  development 
strategy  contributes  to  cost  con¬ 
trol  through  reduction  of  uncer¬ 
tainty.  Interestingly,  programs 
that  included  prototyping  had  a 
relatively  higher  cost  growth.  This 
result  may  be  due  in  part  to  the 
timing  of  the  prototype  phase 
within  the  context  of  the  overall 
program  schedule,  since  earlier 
prototyping  makes  data  available 
earlier,  thus  potentially  affecting 
the  baseline  cost  estimate  at  the 
time  of  EMD  start.  Our  results  are 
consistent  with  this  notion.  It  may 
also  be  true  that  prototyping  was 
conducted  for  programs  with  rela¬ 
tively  higher  degrees  of  technical 
uncertainty,  a  hypothesis  that 
deserves  further  exploration. 
(Drezner  et  al.,  1993,  p.  51) 

Given  that  DoD  prescribes  risk  reduc¬ 
tion  such  as  prototyping  for  riskier  pro¬ 
grams,  RAND’s  assessment  rings  true. 
While  the  prototyping  probably  does 
significantly  reduce  risk,  it  apparently 
does  not  reduce  it  to  the  extent  that  would 
make  a  prototyped  program  have  less  cost 
growth  than  a  non-prototyped  program. 

HAVAIR  Study 

Naval  Air  Systems  Command 
(NAVAIR)  presents  its  most  recent  study 
on  cost  growth  at  the  2001  DoD  Cost 
Analysis  Symposium,  corroborating 
some  of  the  results  of  previous  studies. 
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Here  again,  SAR  data  are  used.  As  part 
of  their  analysis,  they  explore  the  pos¬ 
sible  need  for  “cohort  tracking”  when 
analyzing  cost  growth  (Dameron,  Pullen, 
Summerville,  Coleman,  &  Snead,  2001). 

By  cohort  tracking ,  the  NAVAIR 
team  refers  to  the  grouping  of  cost 
growth  according  to  certain  program¬ 
matic  characteristics  that  relate  to  com¬ 
mon  patterns  of  cost  growth.  The  team 
divides  program  cost  growth  into  five 
categories  or  cohorts  —  RDT&E  cost 
growth  for  programs  with  a  planning 
estimate  (PE)  and  a  development  esti¬ 
mate  (DE);  RDT&E  cost  growth  for  pro¬ 
grams  with  a  DE  only;  procurement  cost 
growth  for  programs  with  a  PE,  a  DE, 
and  a  production  estimate  (PdE);  pro¬ 
curement  cost  growth  for  programs  with 
a  DE  and  a  PdE  only;  and  procurement 
cost  growth  for  programs  with  a  DE 
only  (Dameron  et  al.,  2001). 

The  three  different  estimates  (PE,  DE, 
and  PdE)  are  the  baseline  estimates  that 
correspond  to  each  of  the  major  decision 
points  in  Milestone  A,  B,  and  C  respec¬ 
tively.  NAVAIR  uses  the  five  cohorts  con¬ 
sisting  of  the  different  types  of  estimates 
to  categorize  the  cost  growth,  because  the 
use  of  those  mixes  of  cost  estimates  relate 
to  different  types  of  program  structures, 
which  might  represent  distinct  populations 
with  distinct  cost  growth  patterns. 

After  looking  at  318  DoD  programs, 
the  cohort  study  results  show  that  the 
PE  and  DE  cohort  has  an  average  of  30 
percent  RDT&E  cost  growth;  the  DE- 
only  cohort  has  an  average  of  25  per¬ 
cent  RDT&E  cost  growth;  the  PE,  DE, 
and  PdE  cohort  has  an  average  of  35 
percent  procurement  cost  growth;  the 
DE  and  PdE  cohort  has  an  average  of 
25  percent  procurement  cost  growth;  and 


the  DE-only  cohort  has  an  average  of  15 
percent  procurement  cost  growth.  The 
sample  sizes  are  25,  140,  6,  53,  and  94 
respectively  (Dameron  et  al.,  2001).  The 
NAVAIR  group  indicates  that  the  “results 
are  tentative,”  but  their  findings  suggest 
differences  in  cost  growth  from  one  co¬ 
hort  to  another. 

In  particular,  they  point  out  that,  in  then- 
study,  “programs  with  a  Program  Defini¬ 
tion  and  Risk  Reduction  (PDRR)  phase 
have  more  growth”  (Dameron  et  al., 
2001,  p.  11).  The  pur¬ 
pose  of  PDRR  is  to  re¬ 
duce  risk,  so  programs 
with  PDRR  naturally 
have  a  lot  of  uncer¬ 
tainty,  and  quantifying 
the  costs  of  such  a  pro¬ 
gram  should  be  more 
difficult  than  for  less 
risky  programs.  In  ad¬ 
dition,  the  natural  cor¬ 
relation  between  pro¬ 
grams  with  a  PDRR 
phase  and  programs 
with  a  prototyping  ef¬ 
fort,  leads  one  to  expect  similar  results  as 
the  two  relate  to  cost  growth.  Finally,  pro¬ 
grams  with  a  PDRR  phase  tend  to  have  a 
longer  RDT&E  funding  year  stream  than 
those  without  PDRR.  This  meshes  with 
RAND’s  finding  that  longer  programs 
tend  to  have  higher  cost  growth.  Thus, 
consistency  exists  in  the  findings  show¬ 
ing  that  programs  with  risk  reduction  ef¬ 
forts  tend  toward  higher  cost  growth. 

The  NAVAIR  study  also  analyzes  cost 
growth  correlations  between  program 
phases  and  between  the  RDT&E  and  pro¬ 
curement  appropriations.  The  study 
finds  a  significant  correlation  between 
RDT&E  cost  growth  in  the  PDRR  phase 


"By  cohort 
tracking ,  the 
NAVAIR  team 
refers  to  the 
grouping  of  cost 
growth  according 
to  certain 
programmatic 
characteristics 
that  relate  to 
common  patterns 
of  cost  growth." 
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and  RDT&E  cost  growth  in  the  EMD 
phase  and  also  finds  “significant  corre¬ 
lation  between  procurement  growth 
during  the  EMD  and  production  phases” 
(Dameron  et  aL,  2001,  p.  14).  Finally, 
the  study  uncovers  a  significant  corre¬ 
lation  between  appropriations  such  that, 
during  EMD,  when  the  RDT&E  appro¬ 
priation  experiences  cost  growth,  so 
does  the  procurement  appropriation 
(Dameron  et  ah,  2001). 

As  a  third  area  of  study,  the  NAY  AIR 
group  analyzes  how  program  size  affects 
cost  growth.  Unlike  the  BMDO  and 
RAND  studies,  the 
NAY  AIR  team  performs 
statistical  comparisons 
that  reveal  that  high  and 
low  dollar  programs 
have  identical  distribu¬ 
tions  despite  “a  trend  of 
more  high  end  extrema 
in  the  smaller  size 
classes”  (Dameron  et  ah, 
2001,  p.  21).  To  explain  the  difference  in 
the  extrema,  they  reason  that,  “high  risk 
programs  may  be  terminated  earlier  if 
large,  but  tolerated  if  small”  (Dameron  et 
aL,  2001,  p.  21).  Thus,  inferential  statis¬ 
tics  suggests  no  difference  in  the  overall 
cost  growth  based  on  size. 

Next,  NAY  AIR  studies  the  effects  of  the 
era  in  which  an  acquisition  terminates. 
The  team  uses  “DoD  programs  with  DE 
only  from  the  RAND  93  dataset,  NAY  AIR 
programs  with  DE  only  from  the  SAR  00 
dataset,  and  NAYAIR  programs  with  DE 
only  from  the  Contract  dataset  (RDT&E 
only)”  (Dameron  et  aL,  2001,  p.  23).  Thus, 
they  use  three  separate  data  sets,  two  of 
their  own  compilation  and  the  RAND  93 
dataset.  The  group  studies  the  effects  of 
two  eras  —  pre-1986  and  post- 1986. 


"NAVAIR's 
results  suggest 
that  acquisition 
changes  since  the 
end  off  the  Cold 
War  lead  to  less 
cost  growth/' 


They  choose  1986  as  a  dividing  point, 
because  that  year  marks  the  last  year  of 
the  Reagan  arms  buildup  (Dameron  et  al., 
2001).  The  team  performs  f-tests  to  deter¬ 
mine  if  the  two  eras  differ  statistically. 
They  find  the  following  results; 

•  RAND  93:  The  means  of  programs 
through  1986  and  those  after  1986  did 
show  a  statistical  difference  for 
RDT&E,  but  not  for  procurement. 

•  SAR  00:  The  means  of  programs 
through  1986  and  those  after  1986 
did  show  a  statistical  difference  for 
procurement,  but  not  for  RDT&E. 

•  Contract:  The  means  of  programs 
through  1986  and  those  after  1986  did 
not  show  a  statistical  difference  for 
RDT&E.  (Dameron  et  al.,  2001,  p.  31) 

The  team  concludes  that  their  “analy¬ 
sis  supports  a  decline  in  cost  growth  fac¬ 
tor  (CGF)  over  time”  (Dameron  et  al., 
2001,  p.  32).  They  mention  that  these 
results  differ  from  previous  studies  per¬ 
haps  because  of  differences  in  the  num¬ 
ber  of  data  points  or  dates  of  era  division 
(Dameron  et  al.,  2001). 

NAYAIR’ s  results  suggest  that  acqui¬ 
sition  changes  since  the  end  of  the  Cold 
War  lead  to  less  cost  growth.  Although 
difficult  to  pinpoint  the  reason  for  this 
decline  in  cost  growth,  logic  leads  to 
candidate  scenarios.  The  draw-down  in 
military  spending  after  the  Cold  War  pro¬ 
duces  an  environment  in  the  government 
where  meeting  cost  goals  becomes  more 
important  for  program  survival.  So  per¬ 
haps  the  ensuing  emphasis  on  better  es¬ 
timating  improves  base-line  estimates 
from  which  cost  growth  is  measured. 
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Improvements  in  the  estimating  profes¬ 
sion  might  also  play  a  part  in  improv¬ 
ing  base-line  estimates. 

NAVAIR  also  studies  differences  be¬ 
tween  commodities  and  their  relation  to 
cost  growth.  The  team  looks  at  all  three 
databases,  but  limits  the  data  to  20  RAND 
93  programs,  1 1  S AR  00  programs,  and 
21  contract  data  programs.  They  conclude 
that  missile  programs  experience  higher 
cost  growth  during  RDT&E  than  either 
electronic  or  aircraft  programs.  This  result 
suggests  that  cost  growth  may  vary  by 
commodity. 

The  NAVAIR  team  further  compares 
RDT&E  cost  growth  in  small  programs 
(less  than  one  billion  dollars  in  RDT&E) 
as  portrayed  through  the  SAR  2000  data 
versus  the  NAVAIR  contract  database. 
Their  analysis  shows  that  the  results  from 
the  two  databases  do  not  significantly  dif¬ 
fer  (Dameron  et  al.,  2001).  They  conclude 
that  potential  exists  to  use  either  database 
to  study  cost  growth.  Intuitively,  cost 
growth  from  a  contract  only  perspective 
mirrors  that  of  an  overall  program  cost 
perspective,  because  the  vast  majority  of 
program  dollars  apply  to  contracts. 

Christensen  and  Tempun  Study 

David  Christensen  and  Carl  Templin  re¬ 
search  cost  growth  using  the  Defense  Ac¬ 
quisition  Executive  Summary  (DAES) 
database  and  arrive  at  potentially  useful 
findings  in  the  search  for  predictors  of  cost 
growth.  The  DAES  database  contains  con¬ 
tractor  information  organized  according 
to  the  rules  of  Earned  Value  Management, 
a  process  by  which  the  government 
monitors  the  cost  and  schedule  perfor¬ 
mance  of  contracts  against  baseline  fig¬ 
ures  (Christensen  &  Templin,  2000). 


The  researchers  consider  “hundreds 
of  DoD  defense  acquisition  contracts 
from  1975  through  1998”  in  a  hypoth¬ 
esis  testing  scenario  focused  on  the  na¬ 
ture  of  management  reserve  (MR)  bud¬ 
gets  (Christensen  &  Templin,  2000). 
DoD  characterizes  an  MR  budget  as  “a 
reserve  for  uncertainties  related  to  in¬ 
scope  but  unforeseen  work”  (DoD, 
1997,  p.  12).  MR  budgets,  because  they 
represent  the  contractors’  assessment  of 
risk  for  acquisition  programs,  can  pro¬ 
vide  useful  insight  into  the  overall  risk 
assessment  that  DoD  uses  in  its  bud¬ 
geting  process. 

Christensen  and  Templin  recognize 
that  many  factors  affect  the  develop¬ 
ment  of  a  contractor’s  MR  budget,  and 
that  the  “achievability  of  a  budget  de¬ 
pends  on  how  the  bud¬ 
gets  are  established” 

(Christensen  &  Templin, 

2000,  p.  195).  Thus, 
overruns  can  vary  de¬ 
pending  on  factors 
such  as  differing  meth¬ 
ods,  abilities,  and  mo¬ 
tivations  of  those  who 
set  the  MR  budgets 
(Christensen  &  Templin, 

2000).  A  1998  survey  of 
300  DoD  risk  analysis 
professionals  (U.S.  Aerospace  Cost  Risk 
Analysis  Survey,  2000)  supports  this 
statement  by  displaying  the  following 
variety  of  perspectives  on  risk  analysis 
extant  within  government  and  contrac¬ 
tor  circles. 


"They  [the 
team]  conclude 
that  missile 
programs 
experience 
higher  cost 
growth  during 
RDT&l  than 
either  electronic 
or  aircraft 
programs." 


•  27  percent  of  analyses  perform  the 
risk  assessment  separately  from  the 
cost  estimate. 
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*  26  percent  of  program  managers  do 
not  accept  risk  assessment  at  all,  not 
even  slightly . 

*  32  percent  of  the  risk  assessments  do 
not  involve  finance  or  estimating, 

*  38  percent  of  cost  risk  analysts  have 
received  no  training,  either  formal  or 
informal, 

*  44  percent  of  risk  ranges  are  intuitive 
judgments,  without  historical  data  or 
guided-survey. 

*  69  percent  of  variable  distributions  are 
triangular, 

*  1 8  percent  of  unfavorable  assessments 
are  ignored,  as  managers  stay  the 
course . 

In  addition,  Christensen  and  Templin 
(2000)  note  that  contractors  should  pro¬ 
vide  greater  MR  budgets  for  riskier 
projects.  The  authors  characterize  the 
development  phase  of  acquisition  as  more 
uncertain  than  the  production  phase,  and 
they  characterize  price  contracts  as  more 
uncertain  than  cost-reimbursement  con¬ 
tracts  (Christensen  &  Templin,  2000), 
From  this  awareness  of  the  diversity  of  risk 
analysis,  Christensen  and  Templin  per¬ 
form  hypotheses  testing  to  realize  the 
following  results: 

The  amount  of  an  MR  budget  is 
sensitive  to  contract  category 
(cost-reimbursable  versus  fixed- 
price),  and  the  managing  service. 
With  regard  to  contract  category, 
the  median  MR  percent  on  fixed- 


price  contracts  is  significantly 
greater  than  the  median  MR  per¬ 
cent  on  cost  reimbursable  con¬ 
tracts.  This  is  consistent  with  the 
expectation  that  contracts  with 
more  risk  to  the  contractor  have  a 
larger  MR  budget.  We  do  not 
know  why  MR  budgets  differ 
across  the  three  services.  Possible 
explanatory  factors  include  differ¬ 
ences  in  the  weapon  systems  pur¬ 
chased  by  each  service,  and  the 
contractors  that  build  the  systems. 
(Christensen  &  Templin,  2000, 
p.  204) 


With  regard  to  the  acquisition  phase, 
the  researchers  do  not  find  that  the  MR 
budget  differs  between  production  and 
RDT&E  contracts  (Christensen  &  Temp- 
lin,  2000).  Christensen  and  Templin 
(2000)  shed  light  into  the  way  that  con¬ 
tractors  manage  risk  through  MR  budgets. 
The  relationship  between  contract  type 
and  the  MR  budget  stands  to  reason,  since 
certain  contract  types  place  more  risk  on 
the  contractor  than  others.  The  MR  bud¬ 
get  insensitivity  to  acquisition  phase 
differs  from  the  government's  perspective 
that  RDT&E  efforts  are  more  risky  than 
procurement  efforts;  this  difference  reem¬ 
phasizes  the  importance  of  using  contract 
type  to  instigate  contractor  behavior  that 
best  advances  government  objectives.  The 
sensitivity  to  managing  service  proves 
enigmatic.  While  possible  that  significant 
differences  exist  in  the  way  each  service 
manages  its  contracts,  it  may  be  that  some 
other  variable  or  variables,  highly  corre¬ 
lated  with  managing  service  triggers  this 
sensitivity. 
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Cost  Growth  Specific  to 
the  Aerospace  Industry 


Eskew  Study 

To  find  the  true  rate  of  cost  growth  of 
fighter  aircraft  over  time,  Henry  Eskew 
runs  a  multiple  linear  regression  of  17 
tactical  aircraft  from  1950  through  1980 
(Eskew,  2000).  He  normalizes  his  data  for 
production  quantity  by  using  the  estimated 
100th  production  unit  cost,  and  he  nor¬ 
malizes  his  data  for  inflation  by  applying 
the  appropriate  DoD  inflation  indices  to 
convert  his  data  to  constant  year  (CY) 
1990.  Using  the  logarithm  of  cost  as  his 
response  variable,  he  finds  weight,  speed, 
production  rate,  and  time  as  statistically 
significant  predictor  variables  that  explain 
“more  than  90  percent  of  the  variation  in 
cost”  (Eskew,  2000,  pp.  211-212).  He  also 
determines  that,  as  a  sole  predictor,  time 
explains  about  40  percent  of  the  cost 
variation  (Eskew,  2000). 

Although  useful  as  a  literature  review, 
one  must  note  some  limitation  of  the 
Eskew  study’s  applicability.  First,  the 
study  looks  at  a  limited  amount  of  data 
from  a  limited  perspective.  It  only  con¬ 
siders  tactical  aircraft  in  its  search  for 
predictors,  and  it  only  has  17  data  points. 
In  addition,  this  research  lacks  currency, 
spanning  the  period  from  1950  through 
1980,  and  seeks  to  explain  cost  growth  as 
overall  increases  in  unit  cost  measured 
from  previous  programs  overtime  (Eskew, 
2000).  Most  of  the  research  we  consider 
heretofore  considers  cost  growth  within  a 
single  program  over  a  much  shorter  time 
period. 

Dr.  Eskew  (2000)  seeks  to  dispel  the 
myth  that  “no  systematic  relationship 
exists  between  the  characteristics  of  an 
aircraft  program  and  the  length  of  its 


development  cycle”  (Eskew,  2000,  p. 
210).  He  uses  the  same  normalization 
techniques  mentioned  earlier  for  inflation 
and  quantity;  however,  | 
he  includes  different 
aircraft,  adding  non¬ 
tactic  al  fixed  wing  air¬ 
craft,  and  removing 
non-fixed  wing  air¬ 
craft  (Eskew,  2000). 

The  results  of  his  17 
data-point  regression 
show  that  unit  flya¬ 
way  cost  predicts  ap¬ 
proximately  60  per¬ 
cent  of  the  variance  in 
the  length  of  the  de¬ 
velopment  program: 
this  predictive  ability 
increases  to  70  percent  when  a  dummy 
variable  is  added  indicating  whether  or 
not  a  program  has  inherited  a  significant 
amount  of  technology  from  a  previous 
program  (Eskew,  2000).  Overall,  the 
Eskew  study  highlights  a  correlation  be¬ 
tween  aircraft  physical  and  functional 
characteristics  and  production  costs,  and 
between  program  schedule  and  produc¬ 
tion  costs. 


fv 


If: 


"Overall,  Hie 
Eskew  study 
highlights  a 
correlation 
between  aircraft 
physical  and 
functional 
characteristics 
and  production 
costs,  and 
between  program 
schedule  and 
production  costs." 


IDA  Study 

The  Institute  for  Defense  Analyses  (IDA) 
performs  an  analysis  on  cost  and  schedule 
growth  of  tactical  missiles  and  tactical  air¬ 
craft  in  1994  with  the  goal  of  finding  pat¬ 
terns  of  cost  growth  and  the  reasons  for 
the  cost  growth  (Tyson,  Harmon,  &  Utech, 
1994).  Within  the  group  of  20  tactical  mis¬ 
siles  investigated,  the  IDA  group  finds  that 
only  two  stay  within  their  schedule,  with 
one  program  slipping  by  as  much  as  180 
percent.  They  also  find  that  only  two  pro¬ 
grams  stay  within  budget,  while  the  two 


87 


Defense  Acquisition  Review  Journal — January-April 2004 


worst  performers  exceed  their  budgets 
by  a  factor  of  two  (Tyson  et  al.,  1994). 
The  researchers  of  IDA  examine  the 
characteristics  of  the  programs  with  the 
highest  and  lowest  schedule  and  cost 
growth.  From  their  study,  they  find  that: 

[Missile]  programs  that  employed 
a  high  degree  of  concurrency,  that 
had  to  be  dual-sourced  for  tech¬ 
nical  reasons  or  that  were  dual- 
sourced  at  less  than  full  rate,  had 
high  cost  growth.  In  one  case,  the 
threat  of  competition  appeared  to 
reduce  costs.  (Tyson  et  aL,  1994, 

P-  S-2) 

The  results  from  aircraft  programs  do 
not  vary  as  much.  The  authors  suggest 
closer  management  scrutiny  and  “protec¬ 
tion  from  schedule  stretch”  as  reasons  for 
the  more  consistent  cost  growth  in  aircraft 
programs  (Tyson  et  al,  1994,  p.  S-2).  Two 
aircraft  programs  suffer  elongated  produc¬ 
tion  schedules,  but  do  not  experience  high 
production  cost  growth.  The  authors  theo¬ 
rize  that  generally  extending  production 
incites  cost  growth;  however,  In  these 
cases  the  existence  of  other  DoD  contracts 
cushions  the  impact  of  the  adjusted  sched¬ 
ules,  The  authors  identify  the  F/A-18  as 
the  program  with  the  highest  cost  growth. 
They  theorize  that  late  engineering 
changes  Incite  the  high  cost  growth  (Tyson 
et  al,  1994). 

The  study  considers  whether  modifica¬ 
tion  programs  have  lower  cost  growth  than 
new  start  programs.  The  one  aircraft  in 
their  sample  that  Is  a  modification  does 
in  fact  experience  low  cost  growth.  The 
team  finds  that  missile  modification  pro¬ 
grams  vary  greatly  in  the  cost  growth  they 


experience.  They  cite  the  fact  that  most 
missile  modifications  affect  the  expensive 
guidance  and  control  system  of  the  missile 
as  a  possible  reason  for  this  inconsistency 
(Tyson  et  al.,  1994). 

The  researchers  further  find  that  the 
urgency  of  the  program,  the  difficulty  of 
the  technology,  the  amount  of  concur¬ 
rency,  and  the  degree  of  testing  all  seem 
to  affect  cost  growth  (Tyson  et  al.,  1994), 
From  these  results,  the  IDA  researchers 
discover  a  relationship  between  cost 
growth  and  schedule  growth  in  both  the 
development  and  the  production  phases 
(Tyson  et  al.,  1994).  They  find  that  quan¬ 
tity  increases  during  development  largely 
drive  development  schedule  growth.  The 
authors  mention  “the  need  to  produce 
more  items  for  testing  than  planned”  as 
the  reason  for  the  increase  in  quantity 
(Tyson  et  al.,  1994,  p,  S-6). 

It  is  not  clear  whether  failed  tests  drive 
the  need  to  produce  more  test  units  or  sim¬ 
ply  uncertainty  in  the  planning  process 
drove  the  need  to  produce  more  test  units. 
Either  way,  one  can  draw  a  link  back  to 
technical  risk  from  the  need  for  more  test 
units.  The  study  also  finds  that  whether  a 
missile  is  an  intercept  missile  and  the 
length  of  the  original  schedule  prove  use¬ 
ful  predictors  of  development  schedule 
growth. 

Lastly,  the  IDA  study  adds  depth  to  the 
study  of  cost  growth  by  using  multiple 
regression  to  arrive  at  predictive  formu¬ 
las,  Using  multiple  regression  brings  with 
it  the  benefit  of  taking  into  account  the 
multi-dimensional  interactions  that  inde¬ 
pendent  variables  have  with  dependent 
variables  that  can  prove  misleading  in 
simple  linear  regression  analysis  and 
hypothesis  testing. 
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RAND  Study  (2001) 

In  support  of  the  Joint  Strike  Fighter 
(JSF)  program,  RAND  studies  the  effect 
of  competition  on  the  amount  of  cost 
growth  that  occurs  in  both  the  RDT&E 
and  procurement  budgets  (Birkler,  Graser, 
Arena,  Cook,  Lee,  &  Lorell,  2001).  The 
researchers  analyze  14  programs  that  use 
competitive  strategies  and  44  programs 
that  do  not  (Birkler  et  al.,  2001).  They 
find  that  “the  results  are  mixed  and  the 
differences  between  the  competitive  and 
noncompetitive  development  [and  pro¬ 
curement]  cost  growth  factors... are  not 
statistically  significant  at  the  10-percent 
level”  (Birkler  et  al.,  2001,  p.  80). 

These  results  are  potentially  mislead¬ 
ing,  however,  in  that  they  represent  a 
program- wide  look,  rather  than  a  study  of 
individual  contracts.  In  other  words,  a  pro¬ 
gram  might  be  competitively  awarded 
initially,  but  at  a  certain  point  in  the  string 
of  contracts  that  make  up  a  program  one 
can  make  the  case  that  a  competitive  en¬ 
vironment  no  longer  exists.  Perhaps  a 
study  that  compared  individual  contracts 
(rather  than  entire  programs)  might  exact 
different  results. 

Conclusion 


In  this  article,  we  document  many  stud¬ 
ies  that  query  different  databases  using 
various  statistical  methods  in  the  effort  to 
explain  cost  growth  in  DoD  acquisition. 
We  consider  studies  of  overall  DoD 
acquisition  as  well  as  studies  that  focus 
on  a  particular  industry  within  the  DoD 
acquisition  landscape.  Further  research  is 
currently  being  conducted  to  examine/ 
investigate  how  best  to  model  cost  growth 


and  which  predictor  variables  should  be 
included  in  such  a  model.  From  these, 
appropriate  statistical  tools  and  method¬ 
ologies  could  then  be  made  available  to 
the  cost  community. 

This  article  applies  not  just  to  the  cost 
estimator,  but  also  to  the  entire  integrated 
process  team.  Knowledge  of  these  stud¬ 
ies  should  bring  to  bear  a  better  under¬ 
standing,  a  sharper  focus,  and  a  more 
efficient  approach  to  those  who  seek  to 
study  cost  growth  in  the  future.  It  should 
also  be  clear  to  the  estimator  that  more 
often  than  not,  estimates  will  be  low.  From 
the  many  studies  in  this  article,  the  esti¬ 
mator  should  find  some  insight  that  will 
lead  him  to  a  better  analysis  of  risk  to  com¬ 
pensate  in  part  for  that  tendency  to  under¬ 
estimate.  For  the  rest  of  the  integrated  pro¬ 
cess  team,  the  reality  is  that  cost  overruns 
can  kill  programs. 

As  mentioned  in  the  beginning  of  this 
article,  the  two  sides  of  the  solution  coin 
are:  more  realistic  baseline  estimate  (with 
accompanying  risk  dollars)  and  better  cost 
control.  Many  hindrances  might  stand  in 
the  way  of  achieving  more  realistic  esti¬ 
mates  and  better  cost  control.  Those  hin¬ 
drances  might  be  in  the  form  of  priority 
setting,  manpower  shortages,  lack  of  train¬ 
ing,  budget  cuts,  requirement  creep,  or  any 
number  of  other  issues  both  foreseen  or 
unforeseen.  This  article  focused  on  pro¬ 
viding  decision  makers  in  the  acquisition 
world  with  some  historical  insight  into 
what  research  has  been  done  regarding 
cost  growth.  With  knowing  what  has  been 
in  the  past,  it  provides  a  roadmap  to 
prevent  rework  as  well  as  fine  tune  future 
research  and  promote  ready-to-use 
analytical  tools. 
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LOGISTICS  TRANSFORMATION 

DELIVERING  MATERIEL  READINESS 
TO  THE  ARMY 


From  Collie  Johnson,  Editor-in-chief,  Defense  AT&l 

The  Army's  Office  of  the  Deputy  Chief  of  Staff,  G-4,  released  in 
late  December  2003  an  Army  Logistics  White  Paper,  “ Delivering 
Materiel  Readiness  to  the  Army,  ”  describing  logistics  shortfalls  and 
near-term  priorities. 

“It's  our  strategic  vision ,”  said  Lt.  Gen.  Claude  V.  (Chris) 
Christianson,  Army  G-4.  (<The  paper’s  intent  is  to  provide  clear 
guidance  where  we  want  to  take  Army  logistics  in  the  next  two 
years.  It  is  tied  directly  to  what  we  have  experienced  in  the  past 
few  years.  ” 

For  the  benefit  of  our  readers,  the  paper  is  presented  here  in  its 
entirety.  This  summer,  Defense  AT&L  will  also  be  publishing  an 
interview  with  Christianson  that  will  focus  on  the  logistics  failures 
and  successes  he  observed  while  serving  as  the  principal  Operation 
Iraqi  Freedom  logistics  operator. 


The  Army  G-4  exists  to  deliver  materiel  readiness  to  our  Soldiers — a  task  that 
has  remained  the  same  for  years.  Today’s  operating  environment  has  changed;  we 
are  an  Army  at  War... relevant  and  ready.  Our  most  critical  task  is  to  sustain  the 
combat  readiness  of  our  Deployed  Force  and  to  maintain  the  operational  readiness 
of  the  Current  Force.  The  Current  Force  provides  the  warfighting  readiness  that 
serves  our  nation.  The  Current  Force  must  adapt  to  a  changing  enemy  and  fight  and 
win  decisively  against  any  threat.  Our  fundamental  challenge  within  G-4  is  to 
enhance  our  current  capabilities  while  transforming  Army  Logistics  for  tomorrow. 
We  will  accomplish  this  vital  task  by  focusing  our  efforts  on  four  clear  objectives. 
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This  White  Paper  describes  four  G-4  Focus  Areas  we  will  hold  preeminent 
over  the  next  two  years.  It  addresses  known  shortfalls  in  our  current  structure 
that  require  immediate  action,  and  directly  supports  our  Army’s  transition  to  an 
expeditionary  force  that  is  agile,  versatile,  and  capable  of  acting  rapidly  and 
effectively.  These  Focus  Areas  are  the  Army  G-4’s  highest  priority,  and  we  will 
apply  our  policies,  processes,  and  resources  to  ensure  success. 


Focus  Aria  #1- Connect  Army  Logisticians 


Today’s  Army  Logistician  cannot  see  the  requirements  on  the  battlefield.  Our 
customers  cannot  see  the  support  that  is  coming  their  way.  As  a  result,  we  rely  on 
pushing  support  based  on  our  best  estimate  of  what  we  think  the  Soldier  needs. 
Soldiers  order  the  same  item  several  times  because  they  have  no  confidence  support 
is  on  the  way.  We  will  solve  this  problem  by  connecting  Army  Logisticians.  Army 
Logisticians  will  be  an  integral  part  of  the  joint  battlefield  network  with  satellite- 
based  communications  that  provide  24/7  connectivity  on  demand,  enabling  them  to 
pass  and  to  receive  key  data  from  the  battlefield  to  the  industrial  base.  This  con¬ 
nectivity  will  cover  the  battlefield,  and  it  will  provide  Army  Logisticians  the  agility 
and  flexibility  to  quickly  plug  into  and  unplug  from  a  dedicated  network  with  an 
asynchronous  (stand-alone)  capability. 

The  G-4,  along  with  the  U.S.  Army  Materiel  Command  (AMC)  and  the  U.S. 
Army  Combined  Arms  Support  Command  (CASCOM),  will  work  with  the  Chief 
of  Staff  of  the  U.S.  Army  (CSA)  Task  Force  Network  to  ensure  logistics  commu¬ 
nications  solutions  are  embedded  within  the  Army’s  network  and  will  optimize 
joint  and  combined  operations  in  an  expeditionary  environment.  Our  Enterprise 
Resource  Planning  work  in  Battle  Command  Sustainment  and  Support  System 
(BCS3),  Global  Combat  Support  System-  Army  (GCSS-A),  Logistics  Moderniza¬ 
tion  Program  (LMP),  and  Product  Life-cycle  Management  (PLM+)  are  critical  to 
implementing  fully  this  Focus  Area  from  foxhole  to  factory  to  foxhole.  The  logistics 
common  operating  picture  (LCOP)  will  be  improved  by  this  network  connectivity, 
and  it  will  provide  the  vital  link  in  the  joint  commander’s  ability  to  see  the  force 
and  to  make  decisions  based  upon  accurate,  real-time  logistics  information. 


Focus  Area  #2  -  Modernize  Theater  Distribution 


Today’s  Army  is  not  able  to  respond  rapidly  and  precisely  when  support  require¬ 
ments  are  identified.  We  do  not  have  the  battlefield  distribution  system  that  we 
need.  We  cannot  provide  time-definite  delivery  schedules,  and  we  cannot  effectively 
control  physical  movements  across  the  new  battle  environment.  Effective  theater 
sustainment  rests  solidly  on  the  fundamental  concepts  of  distribution-based  logistics. 
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We  need  a  single  focus  on  the  simple  task  of  guaranteeing  delivery — on  time, 
every  time.  We  must  have  a  distribution  system  that  reaches  from  the  Soldier 
at  the  tip  of  the  spear  to  the  source  of  support,  wherever  that  may  be.  Our 
success  will  be  measured  at  the  last  tactical  mile  with  the  Soldier. 

We  will  build  warfighter  confidence  by  increasing  visibility  and  establishing 
flexible,  responsive  distribution  capabilities.  We  will  not  need  to  store  large  quan¬ 
tities  of  supplies  forward  because  we  will  respond  to  customer  requirements  with 
speed  and  precision.  The  G-4  will  work  with  CASCOM  and  the  U.S.  Transporta¬ 
tion  Command,  the  DoD  distribution  process  owner,  to  develop  this  solution  from 
factory  to  foxhole  in  the  joint  environment.  Along  with  AMC  and  the  Defense 
Logistics  Agency,  we  are  committed  to  enabling  an  effective  distribution-based 
sustainment  process.  We  will  work  with  the  CSA  Task  Force  Modularity  to  develop 
this  objective  in  the  near  term. 


Focus  Area  #3  -  Improve  Force  Reception 


We  have  invested  heavily  over  the  past  10  years  in  improving  our  ability  to 
deploy  rapidly  from  our  continental  U.S.  platforms.  The  strategic  movement  of 
forces  by  Large  Medium  Speed  Roll-On/Roll-Off  (LMSR)  vessels  and  C-17  aircraft 
has  significantly  enhanced  our  capabilities.  However,  we  have  not  invested  at  the 
other  end — in  our  ability  to  receive  forces  in  the  theater.  We  are  hamstrung  by  the 
lack  of  an  organizational  construct  that  focuses  on  joint  theater  opening  tasks. 
Today,  we  build  ad  hoc  support  organizations  to  execute  aerial  and  sea  port  of 
debarkation  operations,  and  we  depend  on  forces  from  several  organizations  to 
establish  the  theater  sustainment  base.  This  process  of  receiving  forces  in  theater 
takes  time,  a  luxury  we  will  not  have  as  the  Army  develops  an  expeditionary 
structure  that  is  capable  of  rapidly  deploying  joint-capable  force  modules. 

In  order  to  effectively  facilitate  the  immediate  operational  employment  and  sus¬ 
tainment  of  the  expeditionary  force  flow,  we  will  design  an  integrated  theater¬ 
opening  capability  that  can  respond  on  extremely  short  notice  and  can  execute 
critical  sustainment  tasks  immediately  upon  entry.  That  theater-opening  capability 
will  not  be  an  ad  hoc  organization.  It  must  be  a  support  organization  that  has 
trained  to  the  task.  It  must  be  enabled  with  the  right  tools  to  succeed,  and  it  must 
have  the  capacity  to  expand  to  meet  theater  growth.  The  critical  operational  tasks 
for  this  organization  include:  (1)  providing  operational  sustainment  command  and 
control  with  reach-back  capability  and  initial  network  visibility;  (2)  conducting 
theater  reception,  staging  onward-movement  and  integration  operations,  to  include 
life  support,  force  protection  and  port  of  debarkation  operations;  and  (3)  sustaining 
forces  in  theater  with  theater  distribution  and  requirements  visibility. 
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Focus  Area  #4  -  Integrate  the  Supply  Chain 


Over  the  past  several  years  the  Army  has  taken  supply  reductions  at  many  levels 
for  various  reasons.  We  changed  Army  policy  several  years  ago  to  reduce  the 
amount  of  items  carried  on  unit  prescribed  load  listings  while  simultaneously  reducing 
stock  levels  in  many  authorized  stockage  lists  across  the  field  army.  Additionally, 
we  took  risks  at  the  strategic  level  by  underfunding  strategic  spares  programs.  The 
cumulative  result  of  these  reductions  is  a  lean  supply  chain  without  the  benefit  of 
either  an  improved  distribution  system  or  an  enhanced  information  system.  As  a 
result,  our  Soldiers  are  at  the  end  of  a  long  line  of  communication  with  reduced 
inventories  and  an  old  distribution  system. 

We  will  view  the  supply  chain  in  a  holistic  manner  to  ensure  we  understand  the 
impact  of  actions  across  the  entire  chain,  not  just  at  a  single  level  or  within  a  single 
Service.  This  joint,  end-to-end  view  is  essential  if  we  are  to  provide  the  kind  of 
support  our  Soldiers  deserve.  The  solution  is  an  enterprise  view  of  the  supply 
chain,  and  an  agency  and  a  Service  integration  of  processes,  information,  and 
responsibilities.  We  are  committed  to  developing  the  Army’s  Enterprise  Solution  to 
the  supply  chain  in  close  coordination  and  alignment  with  DoD’s  Focused  Logistics 
Initiative.  Ultimately,  joint  information  will  be  freely  and  automatically  shared  among 
strategic,  operational,  and  tactical-level  headquarters  and  agencies.  Consumers  and 
logisticians  from  all  agencies  and  Services  will  enter  local  supporting  systems,  plug 
into  the  sustainment  network,  and  be  afforded  end-to-end  joint  total  asset  visibility 
(JTAV).  As  a  result  of  our  Theater  Distribution  efforts,  combatant  commanders  will 
be  capable  of  seeing  inventory  in  motion,  as  well  as  seeing  what  is  available  at 
storage  locations,  and  they  will  be  able  to  rapidly  and  effectively  execute  decisions 
that  meet  their  requirements. 


Conclusion 


We  will  build  confidence  in  the  minds  of  the  combatant  commanders  by  delivering 
sustainment  on  time,  every  time.  We  can  do  that  only  if  we  provide  Army  Logisticians 
the  capability  to  see  the  requirements  every  day  and  to  control  the  distribution  to 
guarantee  precise,  time-definite  support.  Army  Logisticians  will  be  part  of  joint  and 
combined  logistics  processes  that  increase  speed  to  deliver  focused  logistics.  We 
will  integrate  real-time  total  asset  visibility  and  seamlessly  connect  to  the  industrial 
base.  This  will  give  us  an  LCOP  that  will  enable  the  kind  of  end-to-end  control 
that  always  delivers  the  right  support  to  the  exact  location  at  the  precise  time 
needed.  If  we  do  not  connect  Army  Logisticians,  improve  the  capability  of  the 
distribution  system,  modernize  force  reception,  provide  integrated  supply  manage¬ 
ment  and  give  the  joint  force  combatant  commanders  JTAV,  we  will  study  these 
same  lessons  after  the  next  major  conflict.  The  Army  G-4  is  committed  to  ensure 
that  we  will  not  have  to  relearn  these  same  lessons. 
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STUFF  AND  FACULTY  FROM 
DEFENSE  ACQUISITION  UNIVERSITY 
RESPOND  TO  CRITICAL  NATIONAL  MISSION 


Faculty  and  staff  members  from  the  Defense  Acquisition  University  are 
providing  ongoing  support  to  the  Iraq  Program  Management  Office  (PMO), 
a  component  of  the  Iraq  Coalition  Provisional  Authority  (CPA).  The  PMO 
was  commissioned  to  provide  oversight,  management,  and  execution  of  the 
infrastructure  reconstruction  effects  in  Iraq.  Under  intense  pressure  to  ex¬ 
pedite  award  of  the  $18.6  billion  supplemental  appropriation — a  mission 
they  took  on  in  November  2003 — DAU  team  members  Garry  Shafovaloff, 
Linda  Neilson,  Lyle  Eesley,  Larry  “Scoop”  Cooper,  and  Bart  Morrison  have 
devoted  countless  hours  and  contributed  considerable  subject  matter  exper¬ 
tise  to  this  critical  national  mission. 

From  the  inception  of  the  CPA-PMO,  Deidre  Lee,  the  Director  of  De¬ 
fense  Procurement  and  Acquisition  Policy  (detailed  to  the  CPA  for  three 
months  as  Deputy  for  Operations),  looked  to  Garry  Shafovaloff  and  Linda 
Neilson  to  provide  key  leadership  roles;  craft  innovative  acquisition  strat¬ 
egies;  manage  the  coordination  and  integration  of  reconstruction  project 
requirements  with  CPA-Baghdad  and  Office  of  the  Secretary  of  Defense; 
resolve  interagency  development  issues;  respond  to  congressional  inquiries; 
and  coordinate  policy  with  the  White  House.  Eesley  and  Cooper  continue 
to  support  the  PMO  and  Sector  PMO  source  selection  process.  Eesley  also 
serves  as  the  technical  team  chief  for  the  source  selection  team.  Morrison 
continues  to  provide  support  to  the  Construction  Contracting  Team  serving 
as  the  lead  knowledge  management  officer  and  workforce  development 
specialist. 

Army  Sgt.  Maj.  Steve  Gebert,  DAU  South  Region,  was  activated  for 
Operation  Iraqi  Freedom  (OIF),  and  served  as  Kuwait  Aerial  Port  of  De¬ 
barkation  (APOD)  Operations  coordinator  from  March  to  May  2003.  From 
May  to  September  he  served  as  an  action  officer  building  daily  and  updated 
situation  report  briefings  for  the  Central  Command  (CENTCOM)  Regional 
General  Staff,  participated  in  those  briefings,  and  worked  distribution  con¬ 
trol  and  movement  logistics  actions.  Since  returning  from  OIF,  Gebert  has 
been  involved  in  building  an  Operational  Logistics  Case  Exercise  based  on 
his  OIF  experience  for  inclusion  in  LOG-304.  He  has  also  been  working 
on  an  article  identifying  key  logistics  shortfalls,  how  they  were  overcome 
in  the  Iraqi  War,  noting  what  went  well,  and  any  recommendations  for 
systemic  change.  .  .  . 

(continued  on  page  98) 
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During  summer  2003,  a  DAU  team  provided  exceptional  performance 
support  and  rapid  response  to  the  Army's  request  for  immediate  instruc¬ 
tional  assistance  for  soldiers  awaiting  deployment  orders,  with  an  immedi¬ 
ate  need  for  training  in  both  the  CON- 100  arena  and  in  CON-234  (Contin¬ 
gency  Contracting),  Team  members  included  Philip  Deaton,  DAU  South; 
Ronald  Fontenot,  DAU  South;  Debbie  Johnson,  DAU  Midwest;  Air  Force 
Maj,  Michael  McGhee,  DAU  Midwest;  Air  Force  Lt  Col.  Ralph  Mitchell, 
DAU  South;  and  Pamela  Oxendine,  DAU  Midwest.  In  less  than  21  days, 
this  small  team  worked  tirelessly  to  deliver  a  nine-day  targeted  training 
course  that  met  the  learning  objectives  of  both  courses  in  the  constrained 
delivery  time,  while  receiving  high  marks  for  quality  and  delivery  from  the 
warfighters. 

Army  Sgt.  James  W.  Colbert,  a  member  of  the  Video  Services  Depart¬ 
ment,  Operations  Group,  DAU  main  campus,  at  Fort  Belvoir,  Virginia,  was 
tasked  with  a  critical  assignment  in  support  of  the  Iraq  Infrastructure  Re¬ 
construction  Office.  Under  intense  pressure  to  deliver  the  finished  product 
to  soldiers  awaiting  deployment  to  Iraq,  he  edited,  enhanced,  encoded,  and 
published  over  20  hours  of  18  separately  recorded  sessions  of  instruction 
in  Contracting,  Most  notably,  he  did  so  in  only  four  working  days,  receiv¬ 
ing  praise  for  the  quality  of  his  work  and  his  efforts  in  delivering  just-in- 
time  training  to  the  nation’s  warfighters. 
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HONEST  PERFORMANCE  ANALYSIS: 
A  NOT-AIWAYS  MIT  REQUIREMENT 


Over  a  decade  after  Operation  Desert  Storm  and  months  after  Operation  Iraqi 
Freedom,  questions  remain  unanswered  about  Patriot  missile  performance  during 
combat  operations.  The  continuing  dispute  over  claims  about  and  analysis  of  Patriot 
success  (and  failures)  during  Desert  Storm  has  helped  drive  what  seems  likely  to 
be  a  more  robust  analysis  of  Iraqi  Freedom  activity. 

J.  Daniel  Sherman’s  article  on  Patriot  PAC-2  development  in  the  Winter  2003 
Acquisition  Review  Quarterly  failed  to  capture  appropriately  the  validity  issues  in 
analysis  of  the  Desert  Storm  experience.  Understanding  the  dispute  over  Patriot 
claims  and  analysis  suggests  the  criticality  of  robust  analysis  of  operational  perfor¬ 
mance  for  an  acquisition  community  that  is  being  asked  to  focus  on  spiral  devel¬ 
opment  and  acquisition — a  focus  that  implies  the  ability  to  bring  real-world  lessons 
into  programs  in  an  effective  and  efficient  manner. 

To  start  with,  the  PAC-2  article  does  not  put  the  Desert  Storm  experience  in  a 
correct  historical  context.  Sherman  asserts  that  Iraqi  “use  of  its  weapon  of  terror 
by  launching  Scud  missile  attacks”  represented  “the  first  time  that  tactical  ballistic 
missiles  would  be  used  in  hostile  wartime  attacks  on  civilian  populations”  (Sherman, 
2003,  p.  41-42). 

In  fact,  use  of  missiles  during  wartime  goes  back  centuries.  The  British  Army 
used  rockets  to  attack  and  set  ablaze  Boulogne,  France,  in  1806.  During  World  War 
II,  the  Germans  launched  2,952  V-2  ballistic  missiles  at  cities  in  England  and  on 
the  Continent.  (These  missiles  killed  2,724  and  wounded  6,467  in  England  and 
killed  4,152  in  Belgium.)  During  the  Iran-Iraq  War,  the  two  sides  shot  a  total  of 
266  missiles  during  the  “War  of  the  Cities”  (Greenwald,  1995).  Missiles  have  long 
been  used  to  threaten  and  attack  civilians.  Prior  to  Desert  Storm,  there  seemed  to 
be  no  effective  active  defense  against  this  threat.  The  Patriot  seemed  to  offer  a  path 
to  change  this  calculus. 

During  Operation  Desert  Shield,  the  United  States  deployed  Patriot  batteries  to 
Saudi  Arabia  and  Israel.  At  the  end  of  Desert  Storm,  both  within  the  U.S.  military 
and  in  the  public  discussion,  the  impression  was  that  Patriot  had  performed  spec¬ 
tacularly,  effectively  blunting  the  Iraqi  Scud  missile  threat.  Sherman  cites  February 
28,  1991,  estimates  of  successful  Patriot  interception  rates  of  70  percent  in  Saudi 
Arabia  and  40  percent  in  Israel  (Sherman,  2003).  Thus,  despite  the  death  of  28 
Americans  in  a  February  25,  1991  Scud  strike  on  Dhahran,  Saudi  Arabia,  and  other 
deaths,  the  Patriot  performance  was  being  rated  a  stunning  technological  success  in 
the  rapid  adaptation  of  a  system  to  undertake  a  more  demanding  mission  than  that 
for  which  it  was  originally  designed. 
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In  the  12  years  since,  however,  the  open  analysis  of  Patriot  performance  has 
called  into  question  the  conclusion  of  tactical  success.  Questions  have  been  raised 
about  data  collection,  evidence  standards,  and  measures  of  effectiveness.  For  example, 
some  have  challenged  the  standard  for  success. 

The  Army  classified  an  engagement  as  a  success  if  it  could  satisfy 
three  criteria.  (1)  An  engageable  Scud  must  be  present,  (2)  The 
“PATRIOT  must  intercept  the  Scud  (i.e.,  detect,  launch,  and  guide  to 
intercept),'  and  (3)  ‘There  must  be  no  significant  ground  damage” 

The  first  criterion — requiring  that  an  engageable  Scud  be  present  — 
clearly  is  trivial. 

Criterion  2  ...  might  sound  like  a  more  demanding  requirement. 
However,  the  term  “intercept”  does  not  imply  that  the  Scud  was  hit 
or  in  any  way  damaged,  it  only  means  that  at  least  one  Patriot  flew 
out  to  the  general  vicinity  of  the  Scud.  ... 

By  satisfying  criteria  1  and  2,  the  Army  is  in  effect  establishing  that 
a  Scud  was  present  and  that  at  least  one  Patriot  fired  against  it  did 
not  grossly  malfunction.  All  that  is  needed  ...  to  declare  such  an 
engagement  to  be  a  success  is  that  ...  no  significant  ground  damage 
occurred.  ...  Even  in  cases  where  Patriot  radar  tracking  data  indi¬ 
cates  that  the  Scud  would  have  impacted  in  a  completely  uninhab¬ 
ited  area  ...  resulting  lack  of  damage  is  taken  as  positive  evidence 
...  even  if  there  is  unambiguous  evidence  that  a  Scud  warhead  hit 
the  ground  and  exploded,  the  Army’s  methodology  would  still  allow 
the  engagement  to  be  classified  as  a  success.  (Lewis  &  Postal,  2000, 
pp.  385-386) 

Try  applying  this  logic  to  other  systems.  Flak  jackets,  for  example,  are 
overwhelmingly  effective  simply  because  the  vast  majority  of  bullets  fired  on  the 
battlefield  miss. 

This  author  has  detailed  knowledge  of  just  one  Scud  missile  incident— the  February 
16,  1991  attack  against  A1  Jubayl,  Saudi  Arabia.  At  the  time  the  missile  struck,  0207, 
February  16,  the  Patriot  Battery  was  down  for  maintenance.  It  came  back  on  line 
at  0212.  This  maintenance  downtime,  however,  did  not  affect  the  effectiveness 
statistics  as  it  was  excluded  from  the  post-war  analysis. 

For  this  case,  post-war  Patriot  analysis  asserted  that  there  was  no  detection  and 
that  the  missile  impacted  10  miles  north  of  F/2-7  Battery  (F  battery  of  the  2-7 
battalion  (F/2-7)  under  the  11th  Air  Defense  Artillery  (ADA)  Brigade).  The  missile 
landed  “outside  the  defended  area...confirming  that  this  was  a  non-threatening  SCUD.” 
(CAS  Inc,  1992,  as  cited  in  Siegel,  2003,  pp  31-32)  In  fact,  the  missile  landed  just 
150  meters  from  the  pier,  less  than  1000  meters  from  USS  Tarawa  (with  2,793 
sailors.  Marines,  soldiers,  and  civilians  [including  this  author]  aboard),  and  roughly 
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three  kilometers  from  the  battery  site.  The  actual  location  of  the  impact  was  readily 
available  (the  missile  having  been  recovered  nearly  intact  from  the  harbor  bottom) 
yet  the  Patriot  analysis  excluded  this  as  a  case  of  potential  interest  (Siegel,  2003). 

Sherman  asserts  that  “Regardless  of  any  controversy  regarding  the  number  of 
Scuds  that  were  destroyed,  disabled  or  diverted,  the  fact  remains:  Patriots  saved 
many  lives,  both  civilian  and  military”  (Sherman,  2003,  p.  42).  In  terms  of  the  open 
discussion  regarding  Patriot  performance,  the  fact  remains  that  it  is  unproven  whether 
Patriot  saved  any  lives,  whether  civilian  or  military.  In  fact,  some  might  suggest  that 
unwarranted  confidence  in  the  Patriot  system  and  performance  failures  aggravated 
by  operator  processes  may  have  contributed  to  the  death  of  28  Americans  in  the 
February  25,  1991  Scud  attack  against  Dhahran. 

The  fact  of  Desert  Storm  Patriot  performance  is  that  the  contractor  and  the  Army 
attempted— within  a  very  tight  timeline— to  adapt  a  system  from  fixed-wing  air 
defense  to  a  far  more  demanding  mission  of  missile  defense  with  the  penalties  of 
doing  this  with  poorly  understood  parameters  of  the  actual  threat  (Iraqi  Scud)  per¬ 
formance.  They  sought  to  solve  a  very  real  capabilities  gap  through  modifying  the 
existing  system.  In  terms  of  technical  performance,  the  jury  remains  out  12  years 
later,  although  the  question  is  whether  the  Patriots  had  some,  minimal,  or  no  tactical 
capability  against  the  Iraqi  missile  threat  rather  than  the  cited  70  percent  success  rate. 
(A  robust  literature  exists  on  these  issues.  See,  for  example:  Government  Accounting 
Office  (GAO),  1992a;  GAO,  1992b;  Hildreth,  1992;  Lewis  &  Postal,  1993;  Lewis 
&  Postal,  2000;  Postal,  1991;  Postal,  1992;  Stein,  2000;  Sullivan,  1998.) 

On  the  other  hand,  the  Patriot  was  likely  of  great  policy  and  operational  importance. 
Patriot  deployments  in  Saudi  Arabia  and  Israel  reassured  both  of  these  governments 
and  their  people.  The  deployment  to  Israel  might  have  prevented  the  Israelis  from 
striking  back  at  Iraq  and  thus  helped  keep  the  coalition  together.  Thus,  the  Patriot 
performance  during  Operation  Desert  Storm  might  be  a  case  of  a  system  with 
minimal  tactical  effectiveness  but  great  policy  and  operational  effectiveness. 

While  such  details  might  seem  footnotes  to  an  obscure  history,  they  point  to  an 
important  issue  for  the  acquisition  community.  The  challenges  of  the  21st  century, 
the  global  war  on  terrorism,  and  transformation  suggest  that  the  U.S.  military  will 
frequently  go  into  battle  using  non-initial  Operating  Capability  (IOC)  systems,  will 
often  seek  to  employ  these  systems  adaptively  to  confront  emergent  challenges  or 
exploit  new  operational  concepts,  and  will  seek  to  bring  operational  lessons  learned 
back  into  the  procurement  system  as  rapidly  as  possible  to  enhance  future  operational 
capabilities. 

To  do  such  adaptive  procurement  requires  that  analytically  rigorous  and  objective 
analysis  be  applied  to  mission  performance — in  technological,  procedural,  operational, 
and  even  policy  terms.  When  done  right,  this  analysis  should  begin  prior  to  actual 
employment  (potentially  deploying  analysts  with  deployed  units  and  systems)  and 
must  build  on  effective  data  collection  during  operations.  Unfortunately,  beginning 
data  collection  and  analysis  after-the-fact  increases  the  potential  for  skewed  analysis 
and  mistaken  conclusions.  This  analytical  process  and  its  results  should  be  shared 
with  all  stakeholders — the  operators,  procurement  community,  contractors,  and 
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oversight  process  (whether  Do D,  administration,  or  Congressional).  In  addition,  to 
the  extent  that  security  allows,  this  analysis  should  be  available  to  the  larger 
defense  community  to  help  focus  development  priorities  where  merited. 

Following  Desert  Storm,  this  did  occur  with  some  Air  Force  and  Navy  programs 
that  were  perhaps  similarly  experimental  The  Joint  STARS  program,  also  deployed 
for  Desert  Shield/Storm  well  before  official  IOC,  closely  monitored  platform 
performance  during  the  conflict.  The  program  made  a  studied  effort  to  learn  from 
the  experience  in  both  the  technical  and  procedural  realms.  On  the  other  hand,  the 
deployed  experience  was  specifically  excluded  from  the  Operational,  Test  and  Evalu¬ 
ation  (OT&E)  process.  Rather  than  using  the  real-world  experience,  to  accelerate  the 
formal  fielding  of  the  program,  this  experience  was  viewed  by  at  least  some  as  an 
inhibitor  to  program  development  (Fowler,  1998). 

When  Iraq  invaded  Kuwait  in  1990,  the  Navy  Tomahawk  cruise  missile  had 
never  been  fired  in  anger.  From  Desert  Storm  until  today,  the  U.S.  Navy  has  had 
a  team  at  the  Center  for  Naval  Analyses  (key  members  have  included  Richard 
Brody,  Stuart  Dunn,  and  Robbin  Holliday)  to  attempt  to  reconstruct  the  events  and 
effectiveness  of  virtually  every  Tomahawk  strike.  Their  Desert  Storm  work  was  at 
odds  with  original  performance  claims  during  the  heat  of  battle  (Holliday,  1994),  but 
the  analytical  rigor  led  the  Navy  to  accept  and  eventually  embrace  their  work  to 
help  drive  future  program  decisions. 

For  neither  the  Joint  STARS  nor  the  Tomahawk  community  were  all  lessons 
necessarily  pleasant.  Both  communities  undertook,  however,  a  serious  effort  to 
assess  system  capabilities  as  honestly  as  possible  to  enhance  future  performance 
of  what  was  in  one  case  essentially  an  experimental  platform  and,  in  the  other,  an 
untried  weapon  system  when  Iraq  invaded  Kuwait. 

As  far  as  this  author  is  aware,  this  did  not  occur  with  Patriot  performance 
during  Operation  Desert  Storm.  Sherman’s  article  on  PAC-2  development  contin¬ 
ues  a  too-strong  tradition  of  failing  to  evaluate  Desert  Storm  Patriot  performance 
in  an  open,  honest,  forthright,  and  analytically  defensible  manner.  The  assignment 
of  a  Defense  Science  Board  (DSB)  to  examine  Patriot  performance  during  Opera¬ 
tion  Iraqi  Freedom  (Wall,  2003;  Wynne,  2003)  provides  grounds  for  hope  that  this 
error  will  not  be  repeated  with  this  Gulf  War’s  Patriot  employment  analysis.  (For 
initial  open  publication  analysis  of  Operation  Iraqi  Freedom  Patriot  employment, 
see,  for  example,  Ghormley,  2003.) 

The  acquisition  community  should  place  a  high  priority  on  capturing  meaning¬ 
ful  lessons  from  operations — especially  for  systems  that  have  not  yet  achieved 
IOC.  This  experience  should  be  leveraged  to  expedite  fielding  the  most  effective 
systems  in  the  most  appropriate  fashion  amid  the  challenges  of  21st  century  spiral 
development. 


Adam  B.  Siegel 
Senior  Analyst 

Northrop  Grumman  Analysis  Center 
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submissions.  It  is  recommended  that  a 
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the  Department  of  Defense  Acquisition, 
Technology,  and  Logistics  Workforce, 
Topics  for  submission  should  rely  on 
or  be  derived  from  observation  or  experi¬ 
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flect  the  author’s  personal  feelings  or 
thoughts.  Nevertheless,  an  opinion  piece 
should  clearly  express  a  fresh  point  of 
view,  rather  than  negatively  criticize  the 
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version  of  the  manuscript,  especially  ci¬ 
tations  (endnotes  instead  of  footnotes)  into 
the  format  required  in  two  specific  style 
manuals.  The  Defense  ARJ  follows  the 
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author  (date)  form  of  citation.  We  ex¬ 
pect  you  to  use  the  Publication  Manual 
of  the  American  Psychological  Associa¬ 
tion  (5th  Edition),  and  the  Chicago 
Manual  of  Style  (14th  Edition). 
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Bethesda,  MD:  Congressional  Informa¬ 
tion  Service,  Inc. 
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The  Defense  ARJ  is  a  publication  of 
the  United  States  Government  and  as 
such  is  not  copyrighted.  Because  the 
Defense  ARJ  is  posted  as  a  complete 
document  on  the  DAU  home  page,  we 
will  not  accept  copyrighted  articles  that 
require  special  posting  requirements  or 
restrictions.  If  we  do  publish  your  copy¬ 
righted  article,  we  will  print  only  the 
usual  caveats.  The  work  of  federal  em¬ 
ployees  undertaken  as  part  of  their  of¬ 
ficial  duties  is  not  subject  to  copyright 
except  in  rare  cases. 

In  citing  the  work  of  others,  it  is  the 
contributor’s  responsibility  to  obtain  per¬ 
mission  from  a  copyright  holder  if  the  pro¬ 
posed  use  exceeds  the  fair  use  provisions 
of  the  law  (see  U.S.  Government  Printing 
Office,  1994,  Circular  92:  Copyright  Law 
of  the  United  States  of  America,  p.  15, 
Washington,  DC:  Author).  Contributors 


will  be  required  to  submit  a  copy  of  the 
written  permission  to  the  Managing 
Editor  before  publication. 

Copyright  Policy 


We  reserve  the  right  to  decline  any  article 

that  falls  into  these  problem  copyright  cat¬ 
egories: 

•  The  author  cannot  obtain  official  per¬ 
mission  to  use  previously  copyrighted 
material  in  the  article. 

•  The  author  will  not  allow  DAU  to  post 
the  article  with  the  rest  of  the  Defense 
ARJ  issue  on  our  home  page. 

•  The  author  requires  that  unusual  copy¬ 
right  notices  be  posted  with  the  article. 

•  To  publish  the  article  requires  copy¬ 
right  payment  by  DAU  Press. 

Manuscript  Format 


Pages  should  be  double-spaced  and 
organized  in  the  following  order:  title 
page,  abstract,  body,  reference  list, 
author’s  note  (if  any),  and  figures  or 
tables.  Figures  or  tables  should  not  be 
inserted  (or  embedded,  etc.)  into  the  text, 
but  segregated  (one  to  a  page)  following 
the  text.  If  material  is  submitted  on  a  com¬ 
puter  diskette,  each  figure  or  table  should 
be  saved  to  a  separate,  exportable  file  (i.e., 
a  readable  EPS  file).  For  additional  infor¬ 
mation  on  the  preparation  of  figures  or 
tables,  see  CBE  Scientific  Illustration 
Committee,  1988,  Illustrating  Science : 
Standards  for  Publication ,  Bethesda, 
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MD;  Council  of  Biology  Editors,  Inc. 
Please  restructure  briefing  charts  and 
slides  to  a  look  similar  to  those  in  previ¬ 
ous  issues  of  the  Defense  ARJ . 

The  author  (or  corresponding  author 
in  cases  of  multiple  authorship)  should 
attach  to  the  manuscript  a  signed  cover 
letter  that  provides  all  of  the  authors5 
names,  addresses,  telephone  and  fax 
numbers,  and  e-mail  addresses.  The  let¬ 
ter  should  verify  that  the  submission  is 
an  original  product  of  the  author;  that  it 
has  not  been  published  before;  and  that 
it  is  not  under  consideration  by  another 
publication.  Details  about  the  manuscript 
should  also  be  included  in  this  letter:  for 
example,  title,  word  length,  a  descrip¬ 
tion  of  the  computer  application  pro¬ 
grams,  and  file  names  used  on  enclosed 
diskettes,  etc. 

Please  send  us  a  cover  letter;  biographi¬ 
cal  sketch  for  each  author  (not  to  exceed 
70  words);  head  and  shoulder  print(s)  or 
digitized  photo(s)  (saved  at  300  pixels  per 
inch,  at  least  3X2  inches,  and  as  a  TIFF 


file);  prints  of  photos  will  be  accepted  and 
returned  upon  request,  one  copy  of  the 
printed  manuscript;  and  any  diskettes. 
These  items  should  be  sturdily  packaged 
and  mailed  to:  Department  of  Defense, 
Defense  Acquisition  University,  Attn: 
DAU  Press  (Defense  ARJ  Managing 
Editor ),  9820  Belvoir  Road,  Fort  Belvoir, 
VA  22060-5565. 

In  most  cases,  the  author  will  be  noti¬ 
fied  that  the  submission  has  been  received 
within  48  hours  of  its  arrival.  Following 
an  initial  review,  submissions  will  be 
referred  to  referees  and  for  subsequent 
consideration  by  the  executive  editor. 
Defense  ARJ, 

Contributors  may  direct  their  questions 
to  the  managing  editor.  Defense  ARJ ,  at 
the  address  shown  above,  or  by  calling 
(703)  805-3801  (fax:  (703)  805-2917), 
or  via  the  Internet  at  norene.fagan- 
blanch®  dau.mil. 

The  DAU  Home  Page  can  be  accessed 
at:  http://www.dau.mil. 


110 


DAU  Website 

http://www.dau.mil 


Your  On-line  Access  to  Acquisition 
Research ,  Consulting ;  Information, 
and  Course  Offerings 


About  DAU 

•  Mission 

•  President's  Message 

•  Command  Brief 

•  President's  Video 

•  DAU  Catalog 

•  Organization 

•  Regions  and  Campuses 

•  DAU  News 

•  Job  Opportunities 

•  2004-2009  DAU  Strategic  Plan 

•  DAU  Command  Briefing 

•  International  Seminar 

Student  Affairs 

•  Course  Schedule 

•  Office  of  the  Registrar 

•  Apply  for  a  Course 

•  Request  a  Transcript 

•  Free  NDIA  Student  Membership 


e-Learning 

•  Virtual  Campus 

•  Continuous  Learning  Center 

•  Continuous  Learning  Modules 

•  e-Learning  Roadmap 

Resources 

•  Acker  Library 

•  Publications 

•  Virtual  Campus 

•  Continous  Learning  Center 

•  Acquisition  Links 

•  Workforce  Management  Directives 

Acquisition  Support 

•  Acquisition  Specialty  Information 

•  AT&L  Knowledge  Support 
System  (AKSS) 

•  DoD  5000  Resource  Center 

•  Acquisition  Support  Center 

•  International  Seminar 

•  Performance  Based  Services 
Acquisition 


Now  you  can  search  the  DAU  Website  and  our  on-line  publications! 


Ill 


112 


V  .VJ 


cucflftofc'ftnHSi 


Surveying  €«st  Growth 
Mqj  Michael  Greiner.  USAF 
:  Capl  Vines  Sipple,  USAF 
May  Edward  '*10113”  White,  USA.!' 


.  Evaluating  the  Impact  of  Electronic 
yjt:  BrawhwS&yvUms  Lessons  Learned  from 
Three  Caws  at  the  Defense  LogifcUcs  Agency 
' : '■•>. : ■  1  \ ,'ir i. Jasaf.haii ' A .  Ahirdl 

”  ’  tiovv  and  Evaluation 

Affects  Aircraft  Acquisition 
U  Col  fJutKl  iXAIftJwf,  USAF  ,  .  V; 

'A/fen» Year Review  offhG:Vhia«.for, 
Transforming  the  Dcferise  Acquisition  System 
Edward  W.  Rogers 

COE  Robert  R  l.tirniingbism.  USAtRet) 

An  InitiajTLook  at  Technology  and  Institutions 
on  Defense  Industry  Consolidation 
K. '  Et;C«S  John  O.  rWessasdc,  t«AF  >  ' 

'  ‘  Msij  David  R.  King.  USAF  v::?;;'., /G 


ATTENTION 

Military  Officers, 

Defense  Industry, 
Government  Executives, 
University  Professors, 
Graduate  Students! 


CALL  H)R  AUTHORS 


Call  for  Authors 

The  DAU  Press  is  actively  seeking  qual¬ 
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lessons-learned,  tutorials,  and  empirical 
research. 

References  must  be  cited  in  your  bibli¬ 
ography.  Research  must  include  a  descrip¬ 
tion  of  the  model  and  the  methodology 
used.  The  final  version  of  your  manuscript 
must  conform  to  the  Publication  Manual  of 
the  American  Psychological  Association  or  the 
Chicago  Manual  of  Style. 

To  obtain  a  copy  of  ARJ  Guidelines  for 
Authors,  visit  the  DAU  Web  site  {http:// 
www.dau.mil/pubs/arq/arqart.asp).  To  in¬ 
quire  about  your  manuscript’s  potential  for 
publication,  call  the  DAU  Press  at  (703) 
805-3801  or  fax  a  request  to  (703)  805- 
2917,  ATTN:  ARJ  (Norene  Fagan-Blanch);  or 
e-mail  Norene  Fagan-Blanch  at  norene.fagan- 
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We  publish  DoD  AT&L  research  articles  that  involve 
systematic  inquiry  into  significant  research  questions. 
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terest  to  the  DoD  AT&L  community.  You  must  use  a  re¬ 
liable,  valid  instrument  to  provide  measured  outcomes. 
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Publishing  Opportunities  in  Management  and  Marketing. 
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To  change,  add  or  delete  your  subscription  to  Defense  AT&L  magazine  and/or 
the  Defense  Acquisition  Review  Journal  (ARJ),  complete  this  form,  sign  it  and 
mail  it  to  us.  You  can  use  this  single  form  for  both  AT&L  and  ARJ . 

Postal  regulations  require  we  have  an  original  signature  on  each  request  form.  NO 
FAXED  or  ELECTRONIC  SIGNATURES  WILL  BE  ACCEPTED. 

This  is  also  a  good  time  to  change  your  business  address  to  your  home  address.  By 
doing  this,  you  preclude  the  release  of  your  name  and  address  to  those  who  request 
our  mailing  list  under  the  provisions  of  the  Freedom  of  Information  Act  (FOIA). 

Please  do  not  include  your  rank,  grade,  service,  title,  or  other  personal  identifiers. 

Thank  you  for  your  interest  in  the  AT&L  magazine  and  the  ARJ. 


NO  POSTAGE 
NECESSARY 
IF  MAILED 
IN  THE 

UNITED  STATES 


BUSINESS  REPLY  MAIL 

FIRST-CLASS  MAIL  PERMIT  NO.  12  FORT  BELVOIR, VA 
POSTAGE  WILL  BE  PAID  BY  ADDRESSEE 


DEFENSE  ACQUISITION  UNIVERSITY 
ATTN:  DAU  PRESS 
9820  BELVOIR  RD  STE  3 
FORT  BELVOIR  VA  22060-9910 
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